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i Overview of the Program for International Student 
Assessment Young Adult Follow-up Study (PISA 
YAFS) 


The Program for International Student Assessment Young Adult Follow-up Study (PISA YAFS) is a 
follow-up study with students who participated in PISA 2012 in the United States. The study is designed 
to measure how performance on PISA 2012 relates to subsequent measures of outcomes and skills of 
young adults on an online assessment, Education and Skills Online (ESO). PISA is administered in the 
United States every 3 years, and is intended to measure the mathematics, science, and reading literacy 
skills of students who are approaching the end of compulsory schooling, at approximately age 15 when 
the majority of these students are in the 10th grade. The second set of data needed for the PISA YAFS 
study was conducted in 2016, approximately 3.5 years after PISA 2012.' 


PISA data suggest that U.S. students are not as prepared for the global economy as their peers in other 
high-performing countries (Organization for Economic Cooperation and Development [OECD] 2014). 
This, in turn, has generated interest in how students’ skills and experiences at age 15, as measured 
through PISA, relate to their subsequent literacy, numeracy, and problem-solving skills and other 
outcomes such as educational attainment and work experiences. Thus, PISA YAFS was developed to 
explore these relationships. Measuring the skills of PISA students after a lapse of about 3.5 years provides 
a frame for comparing students’ skills at two crucial points: at age 15, as students near the end of 
compulsory education and are beginning to think about their future, and at around age 19, as these 


students start adulthood. 


To measure these students’ subsequent literacy, numeracy, and digital problem-solving skills, the study 
used the Education and Skills Online assessment (ESO), an online assessment developed by OECD. The 
ESO’s design is based on the Program for the International Assessment of Adult Competencies (PIAAC), 
a study of adult literacy, numeracy, and digital problem-solving skills focused on those aged 16 to 65 (see 


https://nces.ed.gov/surveys/piaac/). Like PIAAC, the ESO aims to measure cognitive skills needed for 


successful participation in 21st-century society and the global economy. 


PISA YAFS used the ESO along with a few additional questions on current education participation to 


assess participants in literacy, numeracy, and problem-solving domains, as well as to collect information 


PISA 2012 data were collected in October-November 2012 in the United States. PISA YAFS data were collected in March- 
July 2016. At the time of PISA 2012 data collection, students were between the ages of 15 years 3 months and 16 years 2 
months. At the start of PISA YAFS data collection, some 3.5 years later, the PISA 2012 students were between the ages of 18 
years 8 months and 19 years 7 months. 
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on educational attainment, work experiences, and career intentionality of the PISA 2012 participants. 
These data were collected in 2016 following a successful field test in 2015. To learn more about the ESO, 


visit https://www.oecd.org/skills/ESonline-assessment/abouteducationskillsonline/. 


By examining students’ performance on PISA in relation to their subsequent cognitive and noncognitive 
outcomes in young adulthood, as measured through the ESO, the study provides supporting evidence 
needed to understand how well PISA measures the skills that are considered crucial for successful 
participation in adult life. PISA YAFS also strengthens our understanding of U.S. performance on PISA 
and its implications for U.S. college and career readiness and for the skills of our future workforce. This 
information can, in turn, also be used to inform the further development of PISA and future surveys of 
adult skills. 


1.1 Introduction 


This technical report provides information related to the PISA YAFS data collection with a focus on what 
it purports to measure, data processing and weighting, linking to U.S. PISA 2012 data, and how to use the 
data files. 


This document is organized into eight sections with appendixes, exhibits, and tables listed in the table of 
contents. This report includes bookmarked chapters and is searchable by keyword. This online version is 


508-compliant and includes optimizations for people with access challenges. 


1.2 PISA 2012 


The Program for International Student Assessment (PISA) is a system of international assessments that 
focuses on 15-year-old students. PISA assesses the application of knowledge in mathematics, science, and 
reading literacy to problems within a real-life context (OECD 2014). PISA’s focus on 15-year-olds allows 
countries to compare learning outcomes as students near the end of compulsory schooling and seeks to 
answer the question, “What knowledge and skills do students have at age 15?” Thus, PISA does not focus 
explicitly on curricular outcomes and uses the term “literacy” in each subject area to indicate its broad 
focus on the application of knowledge and skills. PISA also gathers information from students about their 
learning environment, educational experiences, and attitudes toward education. In addition, school 


principals provide information on school context and population. Analyses of PISA data provide 
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information on the relative performance of students and on the differences between student environments, 


attitudes, and experiences within and across countries. 


The majority of the PISA 2012 results provide analogous information for mathematics, science, and 
reading. However, because mathematics was the major domain in 2012, more in-depth information is 
available for mathematics. In 2012 and previous years, all three subjects were assessed primarily through 
a paper-and-pencil assessment. However, additional computer-based assessments in mathematics, 
reading, and problem solving were introduced in 2012, as was a financial literacy assessment, all of which 
were optional for countries. PISA 2012 also collected information on students’ backgrounds, attitudes 
toward mathematics, and learning strategies. School principals provided information on the school’s 
demographics and learning environment. Sixty-five education systems, including the United States, 


participated in PISA. 


The PISA 2012 main study in the United States consisted of four major elements: (1) a 2-hour student 
assessment of reading literacy, mathematics literacy, and science literacy; (2) a student questionnaire that 
required approximately 30 minutes to complete; (3) a school questionnaire to be completed by the 
principal or designee that also required approximately 30 minutes to complete; and (4) a 40-minute 
computer-based assessment that was completed by a subset of students who also took the paper-based 
assessment. The optional financial literacy assessment was included in the main paper-based assessment 
for countries that opted to participate (the United States participated in the financial literacy assessment in 
2012). In 2012, a total of 6,094 U.S. students were assessed and 162 U.S. schools participated. 


The PISA YAFS study utilized the financial literacy component of PISA 2012 to help define the PISA 
YAFS field test and main study samples. Students participating in financial literacy in PISA 2012 were 
defined as the field test sample for PISA YAFS. The remaining PISA 2012 students (those not assessed in 
financial literacy) were defined as the main study sample for PISA YAFS. 


1.3 What PISA YAFS Measures 


PISA YAFS measures both cognitive and noncognitive constructs related to adult-life preparedness, skill- 
use, and achievement in young adulthood. The PISA YAFS data comes from two instruments—the ESO 
assessment and questionnaire and an additional module, called the Learning Experience Questionnaire 
(LEQ), added to supplement the ESO’s background questionnaire (BQ). As the ESO did not have 
information on current educational status and could not be edited to add content, the LEQ questionnaire 


module was developed to more fully realize the aims of PISA YAFS. In particular, the questionnaire 
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items in the LEQ component allowed the examination of the relationship between PISA outcomes, either 
in cognitive performance or student characteristics, and the paths that young adults chose after leaving 
high school. The questionnaire items in the LEQ were selected from questionnaire items in PIAAC that 
were not included in ESO. Exhibit 1 provides the components of the PISA YAFS instrumentation and 
data. More detailed information about the LEQ and ESO content is shown in appendixes A and B, 


respectively. 


Exhibit 1. Components of PISA YAFS 


Learning and 
Educational Career 


So [Uler=) fo) aie-Tale Wey 4116 
Online (ESO) 


Questionnaire (LEQ) 


PISA YAFS 
Data 
Collection 


PISA 2012 Dataset 
PISA YAFS Dataset (cognitive and non- 


cognitive data) 


PISA YAFS data may be merged with the 
PISA 2012 dataset 


The ESO includes a core set of background items, two cognitive assessment blocks, and multiple 
noncognitive questionnaire modules administered to all respondents after the assessments. The 
noncognitive indicators collected through PISA YAFS include categorical and continuous data from the 


LEQ and ESO-noncognitive BQ modules. 


The LEQ module gathered information on: 


7 current education study status (participation; level of degree; area of study); 
a formal education activities; and 
8 nonformal learning activities in the 12 months preceding the study. 
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The ESO noncognitive modules collected information on: 


7 basic demographics; 
7 career interests and intentionality (CID); and 
a subjective well-being and health (SWBH). 


Each of these modules is represented in the PISA YAFS database by item-level and indexed data. See 


appendixes A and B — Questionnaires to view the administered questionnaire items. 


1.4 PISA YAFS Administration 


Planning for the PISA YAFS started prior to PISA 2012 data collection in the United States. As part of 
the PISA 2012 field administration, respondents were asked to voluntarily provide contact information to 
the National Center for Education Statistics (NCES) if they were willing to be contacted in the future for a 
follow-up study. A total of 6,094 respondents participated in PISA 2012 in the United States. More than 


90 percent of these respondents voluntarily provided a completed form after the assessment (table 1). 


Table 1. Contact forms returned from PISA 2012: PISA YAFS 2016 


Status of contact forms Count Percent 
Total PISA 2012 participating students 6,094 100.0 
Returned form 5,675 93.1 
Did not return form 419 6.9 
PISA YAFS field test sample 1,116 18.3 
Returned form 1,063 17.4 
Did not return form 53 0.9 
PISA YAFS main study sample 4,978 81.7 
Returned form 4,612 75.7 
Did not return form 366 6.0 


SP SS a a, Fe a Pi a a a a a a a a a ey Se CP rae 
NOTE: The PISA YAFS field test sample was defined as PISA 2012 participating students selected for financial literacy. The 
PISA YAFS main study sample was defined as PISA 2012 participating students who were not selected for financial literacy. 
SOURCE: U.S. Department of Education, National Center for Education Statistics, PISA Young Adult Follow-up Study (PISA 
YAFS), 2016. 


Since the ESO was administered 3.5 years later (in 2016) to PISA 2012 participants, much time was spent 


maintaining reliable communications. These steps are referred to as tracing, tracking, and recruitment 


herein. Tracing tasks locate the PISA participant while tracking tasks maintain channels of 
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communication with the respondent over time. Recruitment tasks get the respondents to begin and finish 


the survey. 


To aid tracing and tracking, Westat developed a participant website where potential respondents were 
able to update their contact information online. Tracing formally ended once a participant registered on 
the website and then tracking and recruitment formally began. The participant website was also used to 
disseminate updates on the study, monitor participation, distribute incentives, and access the LEQ and the 
ESO. The website registration activity provided a proxy for gauging anticipated response to the PISA 
YAFS instruments. 


The registration for the study website opened in 2014, and over the course of the ensuing year and a half, 
all of the field test sample participants registered, and more than half of the total potential main study 
participants registered. A number of other tracing and tracking activities took place prior to launching 


website registration. These activities are discussed in chapter 3. 


The PISA YAFS field test study was conducted December 2015 — January 2016 with a subset of the PISA 
2012 participants who took the financial literacy assessment. These participants were field test candidates 
because they participated in PISA 2012, provided contact information, and were not slated to be included 
in the PISA 2012 main dataset (these students were included in a separate dataset for financial literacy as 
part of the 2012 data release). The purpose of the field test was to evaluate operational procedures and 
refine the background items. Since the ESO software was administered as an off-the-shelf product, the 
field test was primarily focused on proving that respondents were willing and able to self-administer the 
ESO and LEQ instruments. 


The main study data collection for PISA YAFS then occurred between March and July of 2016. 


Originally scheduled to last 12 weeks, the data collection window was extended to meet minimum 


response thresholds. The overall timeline of these activities is shown in exhibit 2. 
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Exhibit 2. Timeline of contact activities and administration of PISA YAFS 2016 


ACTIVITY 2012-2013 
Instruments finalized 
LEQ 
ESO 
Tracing and Tracking 
Initial keying and verification of contact 
information 
First contact email and reminders 
Hard copy packets sent, teleforms 
collected and keyed 
Launch of participant website 
Contact respondents to register 
Call staff training and telephone follow-up 


Pilot study recruitment and data collection 
Primer email 
Data collection 
Email reminders 


Main study recruitment and data collection 
Primer email 
Data collection 
Participant follow-up by call staff 
Email reminders 


® Activity started. 
® Activity completed. 


2014 2015 2016 
arr | arr2 | atrs | arra | arr: | arr2 | arrs | area | area | arr? | arr | arra | 
= —o ——— 


SOURCE: U.S. Department of Education, National Center for Education Statistics, PISA Young Adult Follow-up Study (PISA YAFS), 2016. 
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1.5 Organization of This Document 


This technical report and user guide is designed to provide researchers with an overview of the design and 
implementation of PISA YAFS, as well as with information on how to access the PISA YAFS data. This 
information is meant to incorporate information presented in OECD publications for the ESO, as the ESO 
was a component of PISA YAFS. Chapter 2 provides information about sampling requirements and 
sampling in the United States. Chapter 3 describes the data collection for PISA YAFS and participation 
rates. Chapter 4 describes weighting and variance estimation, and chapter 5 provides a nonresponse bias 
analysis. Chapter 6 describes population modeling, IRT analysis, and scaling of PISA YAFS scores. 
Chapter 7 provides detail concerning data processing and preparation. Chapter 8 describes the data 
available from both international and U.S. sources, and discusses some special issues in analyzing the 
PISA YAFS data. The noncognitive questionnaire items and respondent-outreach materials are included 


in the appendices. 
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2. Sampling 


The PISA YAFS sample comprised all respondents who participated in the PISA 2012 assessment in the 
United States and who did not take the PISA 2012 financial literacy component.” Of the 4,978 PISA 2012 


participants, 366 did not provide contact information. 


This section describes the selection of the PISA 2012 sample and eligibility criteria and the process of 
translating those criteria to current age and eligibility at the time of the PISA YAFS data collection. This 
section does not include breakouts of the field test sample because it was a convenience sample. Field test 


information is found in chapter 3. 


2.1 Overview of the PISA and PISA YAFS Sample 


PISA seeks to show the overall yield (outcomes) of an education system and the cumulative effects of all 
learning experiences for respondents who are nearing the end of compulsory education. PISA’s focus on 
students at age 15 provides an opportunity to measure broad learning outcomes when most students are 
still required to attend school. Drawing on an age-based sample of students makes comparisons across 


countries somewhat easier than a grade-based sample because years of education vary among countries. 


The PISA YAFS results are nationally representative of the PISA 2012 population of 15-year-olds some 


3.5 years later when the majority of respondents were age 19. 


2.2 Target Population 


For students to be eligible for PISA 2012 in the United States, they must have been born between July 1, 
1996, and June 30, 1997. This means that PISA 2012 students in the United States were between 15 years 
and 3 months and 16 years and 2 months at the time of the assessment (Fall 2012). They must also be 


enrolled in a sampled, participating school, at grade 7 or higher. 


2 Although three U.S. states—Connecticut, Florida, and Massachusetts—participated in PISA 2012 with separate state samples, 
these participants were not part of the PISA YAFS data collection effort. Only students who were part of the U.S. national 
sample were eligible to participate in PISA YAFS. 
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Additional eligibility criteria include English language literacy and ability to access the assessment with 
allowed (but limited) accommodations. For more information about PISA eligibility criteria, exclusion 
rates, and accommodations, see the PISA 2012 technical report (available at 


https://nces.ed.gov/pubsearch/pubsinfo.asp?pubid=2014025). 


Considering PISA 2012 eligibility criteria and knowing that PISA YAFS is a follow-up study conducted 
approximately 3.5 years later, the population that participated in the PISA 2012 assessment, when 
contacted to take PISA YAFS in 2016, was between 18 years 8 months and 19 years 7 months old. Thus, 
since PISA employed a nationally representative sample, the PISA YAFS sample is representative of U.S. 
15-year-olds some 3.5 years later, in 2016, when the majority were age 19. Chapter 4 covers the 
nonresponse adjustment and weighting used by PISA YAFS to account for those students who took PISA 
2012 but did not participate in PISA YAFS. 


2.3 Respondent Definition 


Respondents were contacted to participate in PISA YAFS if they voluntarily returned a student 
information form in PISA 2012. The PISA 2012 core assessment had 4,978 participating U.S. students. 
Of these, 4,612 students (92.6 percent) returned an information form. Respondents in PISA YAFS were 
considered participating if they completed at least one Learning Experience Questionnaire (LEQ) item 
regardless of whether or not they went on to complete any of the Education and Skills Online (ESO) 
noncognitive or cognitive sections. Chapter 3 describes the data collection for the field test and main 


study and response rates. 
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3. Data Collection Activities 


The PISA YAFS ESO assessment was administered in 2016, 3.5 years after the respondents had taken 
PISA 2012. From the end of the PISA assessment in fall of 2012 through March 2015, respondents were 
traced using the contact information they provided after the 2012 assessment (meaning they were located 
via e-mail, by phone, or through a physical address) and tracked so that the study maintained 
communication with respondents through the data collection period via e-mail, hard-copy mailing, 
registration on a project-specific participant website, and phone follow-up. This time prior to data 
collection is referred to as tracing and tracking herein. During tracing and tracking, potential participants 


were asked to log into the system and update their contact information. 


As the data collection window approached, the study launched a recruitment effort utilizing the tracing 
and tracking results. Since the tracing and tracking stages did not include responding to any instruments 
or other tasks along the way, as long as the participant was not marked as an active refusal and the study 
had “working” contact information, recruitment was attempted and participants accessed the instruments 
through the participant website. The PISA YAFS field test, conducted December 2015 — January 2016 
with PISA 2012 financial literacy participants, was primarily focused on proving that the design and 


operations for collecting the data were feasible and robust. 


As noted in chapter 1, the main study data collection for PISA YAFS occurred between March and July 
of 2016. Originally scheduled to be completed in 12 weeks, the data collection window was extended to 


maximize response by an additional 6 weeks. 


This section describes the data collection instruments, tracing and tracking, recruitment, field test and 


main study data collection, and quality control procedures. 


3.1 Data Collection Instruments 


As described in chapter 1, the PISA YAFS data collection utilized two instruments: the Learning 
Experience Questionnaire (LEQ) and the ESO assessment, which participants accessed through the 
participant website. The ESO platform required users to use a single specific browser type (Firefox) and 
to access the assessment from a laptop or PC (i.e., no tablets or cellphones). The LEQ did not have any 
device-type or browser restriction. The LEQ was administered prior to the ESO. The cognitive and 


noncognitive pieces of these instruments are described in the following sections. 
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3.1.1 Cognitive Instruments 


By design, the ESO produces individual scale scores in literacy, numeracy, and problem solving in 
technology-rich environments. The cognitive portion of the ESO is adaptive, so depending on how a 
respondent performs on initial cognitive items, additional reading comprehension components rather than 
problem-solving items may be administered. Therefore, some participants in PISA YAFS will not have 
problem-solving or reading comprehension scores depending on their success with the initially presented 
cognitive items. For more detailed information on the ESO cognitive constructs, refer to the Education & 


Skills Online Technical Documentation, available at https://www.oecd.org/skills/ESonline-assessment/ 


assessmentdesign/technicaldocumentation/ESOnline_ Technical Doc _Chl-8 Oct2016.pdf. In particular, 


chapters 2 and 3 of this document (“Development of the Cognitive Instruments” and “Development of the 
Noncognitive Instruments”) provide more general information on the cognitive and noncognitive 


instruments administered through ESO. 


Unlike PIAAC, the ESO was self-administered and offered only on the computer; there was no paper- 
and-pencil component. As in PIAAC, the ESO items for all three cognitive domains were designed to be 
authentic, culturally appropriate, and drawn from real-life situations that were expected to be important or 
relevant in different contexts. Item contents and questions were intended to reflect the purposes of adults’ 


daily lives across cultures, even if they were not necessarily familiar to all adults in all countries. 


The definition of literacy in the ESO, as in PIAAC, is “understanding, evaluating, using and engaging 
with written text to participate in society to achieve one’s goals and to develop one’s knowledge and 
potential” (OECD 2012). The ESO literacy items included continuous texts (e.g., text in sentences and 
paragraphs); noncontinuous texts (e.g., schedules, graphs, and maps); and electronic texts (including 


hypertext or text in interactive environments, such as forms and blogs). 


In the ESO, as in PIAAC, numeracy is defined as: “the ability to access, use, interpret, and communicate 
mathematical information and ideas, to engage in and manage mathematical demands of a range of 
situations in adult life’ (OECD 2012). The ESO numeracy items involved objects or pictures, text, 
numbers, graphs, and technology-based displays. These items required basic mathematical skills in 
computation, proportions, and percentages; an understanding of measurement concepts and procedures; 
and an ability to work with simple formulas. Respondents also encounter more complex items that 
required using models to predict future needs, as well as an understanding of basic statistical concepts and 


displays. 
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In the ESO and PIAAC, problem solving in technology-rich environments (PS-TRE) is defined as: “using 
digital technology, communication tools, and networks to acquire and evaluate information, communicate 
with others, and perform practical tasks” (OECD 2012). As its name implies, PS-TRE measures skills and 
abilities that are required for solving problems while operating in a technology-rich environment. 
Specifically, it assesses the cognitive processes of problem solving—goal setting, planning, selecting, 


evaluating, organizing, and communicating results. 


Post-data collection, two adjustments were made to the resulting PISA YAFS data collected via the online 
survey tools. First, statistical adjustments (reweighting) were made to the original PISA 2012 weights. 
Nonresponse adjustments (NRAs) were performed on the PISA 2012 student weights so that data 
collected in PISA YAFS are statistically valid for making comparisons with the PISA 2012 U.S. national 
sample data. The NRA process is described in detail in chapter 4, with nonresponse bias analyses 
presented in chapter 4. Second, to improve the utility and statistical power of the ESO cognitive data, the 
ESO scale scores were transformed into plausible values (PVs) using an Item Response Theory (IRT) 
2-parameter logistic model, which provides more accurate measurement of student performance than the 
ESO scale scores.’ The plausible values are multiple imputations which represent a range of abilities for a 
certain group of respondents (e.g., all female respondents) (von Davier, Gonzalez, and Mislevy 2009). 
Respondents are not administered every assessment item so each item has missing responses, though 
these are missing by design. Thus, it is not possible to estimate scores for individual respondents. Instead, 
the results of individual respondents are aggregated to produce a set of scores for groups of respondents. 


A description of the re-scaling methodology and considerations is provided in chapter 6. 


3.1.2 Noncognitive Instruments 


The PISA YAFS noncognitive instruments used were the LEQ, the ESO core-background questions, and 


the ESO additional noncognitive background modules. 


The LEQ instrument was developed after close review of the ESO items determined that certain items 
from PIAAC that were not included in ESO were necessary based on the goal of the study and the 


planned analyses. For example, current education participation in formal education and specific types of 


3 The ESO calculates a score for each individual who completes the literacy, numeracy, and problem-solving assessments. While 
the scores produced by the ESO are designed to place respondents’ achievement on the same scale as PIAAC, the online 
version of the ESO produces scores that are calculated in increments of 10 (e.g., 350, 360, 370) and do not include standard 
errors that take into account measurement error. To allow for more nuanced analyses, the ESO scores collected through PISA 
YAFS were recalibrated to more closely mimic the scores produced by PLAAC—that is, scores anywhere along a scale from 0- 
500 and a standard error associated with that score. See chapter 6 for more details. 
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training/instruction recently received were not captured by the ESO. The LEQ questionnaire can be found 


in appendix A. 


The ESO core-background questions are a set of questions that every participant must complete to get 
access to the cognitive assessment. The responses to these questions impact the scale-score value received 


for each individual test-taker. These questions are included in appendix B. 


The other background modules include Subjective Well-being and Health, Career Interest and 


Intentionality, and Skills. The items from these modules can also be found in appendix B. 


3.2 Data Collection Activities 


At a high level, the steps to collect PISA YAFS data involved three main phases: (1) tracing, or the 
process associated with initially validating participation-provided contact information (tracing the 
participant to their location); (2) recruitment for participating in PISA YAFS, or the period directly before 
data collection when participants were directly asked to register for the study website, then establishing 
and maintaining lines of communication (tracking them over time); and (3) data collection, when 
participants were asked to take the survey. These three phases are described in the following subsections, 


and then for the field test and main study separately. 


The PISA YAFS field test frame consisted of the 1,063 U.S. PISA 2012 participants who took the PISA 
2012 financial literacy assessment and returned the contact form. The PISA YAFS main study frame 
consisted of the 4,612 U.S. PISA 2012 participants who returned the contact forms and did not take the 


PISA financial literacy assessment. 


3.2.1 Tracing and Tracking 


The initial tracing and tracking phase was implemented to establish a communication contact with the 
respondents. E-mail and hard-copy mail were the preferred modes of contact across the tracking and 
tracking, recruitment, and data collection phases of the study. Telephone outreach also proved fruitful 
during recruitment and data collection. The reasons for the preference were the ability to send 
communications en masse, and the relatively low cost as opposed to telephone or household interviewing. 
During the tracing and tracking phase, specific information about when the study would take place was 


not provided, nor were participants solicited to take questionnaires or surveys, prior to data collection. 
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Rather, participants were told about the goals of the PISA YAFS, their importance and role in PISA 
YAFS, and that specific, further details would be provided when available. Along with reaching out with 


study information, month-based birthday cards were sent to participants. 


After identifying the untraceable group of PISA 2012 participants using electronic and hard-copy 
mailings, address updates services were used to identify addresses that were updatable. Of the PISA 2012 
participants that provided contact information, 99 participants were identified as “hard to trace” or 
untraceable because complete contact information was not initially provided, the information was wrong, 
or unavailable via follow-up with schools or a locating service. A participant website was launched after 
the initial mailing actions to maintain contact and allow participants to easily update their contact 
information. Table 2 shows the overall results of the tracing effort. In total, 5,191 individuals (92 percent) 


were located through the mailing and e-mail contact efforts. 


Table 2. Overall summary of PISA YAFS 2016 tracing effort 


PISA 2012 participants 

Percentage 

Response count Number (unweighted) 
Total participants to locate (field test and main study combined) 5,634 100.0 
Hard to trace 99 1.8 
E-mail returned 650 11.5 
Hard-copy packet sent 4,885 86.7 
Packet sent successfully 4,541 80.6 
Total located 5,191 92.1 


nan eee eee ee eee eee EE ——— Se Eee eee eee 
SOURCE: U.S. Department of Education, National Center for Education Statistics, PISA Young Adult Follow-up Study (PISA 
YAFS), 2016. 


Of the total participants, 99 were removed because they could not be traced. E-mails were returned from 
650 participants. The remaining 4,885 participants were sent hard-copy mailings with informational 
materials describing the study, their selection, and assurances that further information would be 


forthcoming. 


The PISA YAFS participant website launched in December 2014. The site provide several functions 


including: 


7 serving as a central location for respondents to access information about the study; 


ry tracking respondent contact information; 
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a allowing respondents to access and complete the survey; and 


im releasing the incentive to the respondent after completing the survey. 


Several actions were initiated to facilitate registration on the site. Like the previous tracing work, 
announcement e-mails were sent to the sampled PISA participants to register on the site. For those 
without verified e-mail addresses or who had indicated that hard-copy mailing was their preferred way to 
receive study information, hard-copy mailings were sent with a letter announcing that the site was open 
and encouraging them to register for the study. Appendix D provides examples of the registration 


materials. 


Beginning in March 2015, a telephone contact effort contacted the sample to encourage them to register 
on the site and continue to update participant contact information. This effort continued through May 
2015. 


3.2.2 Recruitment 


Recruitment involves contacting the respondent, presenting a description of what is involved in 
participating in the upcoming study, inviting the respondent to participate and providing the respondent 
with the necessary information to access the survey. The purpose of the recruitment effort is to maximize 
the number of respondents who complete the data collection phase of the study while maintaining the 
contact that was established during the tracing efforts. Appendix D provides the example contact 


materials. 


Recruitment for the field test began November 2015 with priming e-mails to the e-mail addresses on file. 
The priming e-mail contained information about the survey launch and informed potential participants 
that they would need to use Firefox to access the ESO. The priming e-mail also asked participants to 
update their mailing address so that a cash card could be sent to them via FedEx. Hard-copy packets were 
sent via FedEx that contained an official invitation letter from NCES, and a YAFS information flyer. The 
recruitment yielded 371 registrations for the PISA YAFS field test. 


Recruitment for the main study followed a similar priming e-mail and hard-copy mail approach and began 
in February 2016. The contact was made to the 4,612 U.S. PISA 2012 participants who voluntarily 
returned the contact forms. Follow-up calls by Westat staff were made throughout the data collection 
period to nonrespondents to encourage their participation. The recruitment yielded 2,448 registered users 


for the main study. 
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3.2.3 Data Collection 


Data collection was conducted for both a field test and main study. The data collection periods for both 
the field test and main study were kicked off using primer e-mails, which announced when the survey 
would open and reminded respondents about the activation of the incentive card after completing the 
survey, and hard-copy notifications with redundant information (see appendix D for example materials). 
Along with notifying participants of the upcoming data collection and incentives, specific information 


was provided about the ESO’s Firefox browser requirement and supported device types. 


Throughout the 12-week data collection period, e-mails, hard-copy notices, and telephone reminders were 


utilized at key times to drive response. 


PISA YAFS Field Test Data Collection 


The PISA YAFS field test was conducted December 2015 — January 2016. The purpose of the field test 


was twofold: 


1. To examine the operational stability of the online components (e.g., participant website, 
LEQ, ESO, electronic incentive distribution). 


2. To obtain a general feel for the data (i.e., that normal performance distributions were 
evident, and that data were valid). 


Response rates for the PISA YAFS field test are shown in table 3. In total, 258 respondents responded to 
the PISA YAFS field test survey in some way. There were 213 respondents who completed both the LEQ 
and ESO instruments, an additional 34 respondents who completed the LEQ and part of the ESO, and a 
further 11 respondents who completed only the LEQ. 


Timing data was gathered and examined showing that the variation in response times and the number of 


sessions used by respondents to complete the survey. These timings were within the expected limits for 


completing the survey. 
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Table 3. Status of sampled participants selected for the PISA YAFS field test: 2016 


PISA YAFS participation 
Percentage of 
Number of respondents 
Response status respondents (unweighted) 
Total registered 371 100.0 
Responded to LEQ 265 71.4 
Responded to ESO 247 66.6 
Complete 213 57.4 
Partially complete 34 9.2 
Not started 11 3.0 


i ie a a a a a a aE 
NOTE: LEQ refers to the additional noncognitive survey items in the Learning Educational Career Questionnaire and ESO refers 
to Education and Skills Online assessment. 

SOURCE: U.S. Department of Education, National Center for Education Statistics, PISA Young Adult Follow-up Study (PISA 
YAFS), 2016. 


The PISA YAFS field test was also useful for examining the effectiveness of the messaging and 
communication with respondents during the survey window and provided metrics to use in evaluating 
response patterns over time for the main data collection. A schedule of e-mails and call contact efforts 
was applied at specific points at regular intervals. As the end of the field test window approached, 


messaging was able to create a deadline effect that added a large boost in response. 


PISA YAFS Main Study Data Collection 


The PISA YAFS main study data collection occurred between March and July 2016. As described in 
section 3.2.1, the main study work began with tracing and tracking participants from 2012. In late 2015, 
participants began registering for the PISA YAFS website and then were formally recruited to participate 


whether or not they had registered for the website. 


The main study data collection was originally scheduled to run for 12 weeks. However, given that there 
was no requirement that data collection end at a specific time, the data collection window was extended 


by an additional 2 weeks to maximize participant response. 


Data collection opened on March 7, 2016 and ran through June 15, 2016. The original data collection 
window was to close on May 30 and messaging to respondents who had not completed the survey was 
sent on May 21, 2016. The data collection was extended to June 15 in an effort to maximize participant 


response. 
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Table 4. Status of sampled participants selected for the PISA YAFS main study: 2016 


PISA YAFS participants 

Percentage 
Number (weighted) 

U.S. PISA 2012 student sample (core assessment only) 4,978 
PISA students completing student information forms 4,612 100.0 
PISA YAFS 2016 participants 2,318 50.1 
Participated in LEQ 2,318 50.1 
Participated in ESO 2,008 43.6 
Did not participate in ESO 310 6.6 


SSS a a a ey 
NOTE: Of the 4,612 PISA 2012 participants who returned the contact forms, 2,448 registered on the PISA YAFS recruitment 
website. LEQ refers to the additional noncognitive survey items in the Learning Experience Questionnaire. ESO refers to the 
online assessment known as Education and Skills Online. 

SOURCE: U.S. Department of Education, National Center for Education Statistics, PISA Young Adult Follow-up Study (PISA 
YAFS), 2016. 


In total, 2,318 respondents interacted with the PISA YAFS survey in some way. There were 2,008 
respondents who responded to both the LEQ and ESO instruments, and a further 310 respondents who 
completed only the LEQ. 


3.3 Quality Control 


Several quality control measures were implemented to ensure: (1) respondents could successfully access 
the survey; (2) accurate and valid data were being collected (e.g., confirming identity using PISA 2012 
administrative data); and (3) any issues were promptly identified and fixed (e.g., daily and weekly help 


desk reports). 


A PISA YAFS study help desk was established to answer questions from respondents needing technical 
assistance accessing the PISA YAFS participant website as well as the survey instruments. The help desk 
staff also assisted with answering questions about timing and availability of the survey, as well as 
incentive-related questions about the cash cards. The help desk was reached by both e-mail and a toll-free 


800 line. Formal trainings with outbound callers and help desk staff were conducted. 


An important aspect to online data collection surveys is verifying that those individuals responding are 
the intended respondents. The PISA YAFS online system construction built in specific data-driven checks 
from the PISA administrative data (such as respondent gender and birth date) to verify, as much as 
possible, the authenticity of the respondents. Respondents were asked to provide this information when 


they registered for the study and created their personal login credentials. 
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Weekly progress reports detailing the success of data collection were provided to project staff and the 
National Center for Education Statistics (NCES). The reports provided update of respondent completions 
counts and percentages. This report also included overall participation outcomes—number of complete 


surveys with both LEQ and ESO, LEQ only or ESO only, and reported overall rates of participation. 


In addition to the weekly reports, standing weekly project meetings were attended by project staff and 


NCES to review progress and discuss issues. 
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4. Weighting and Nonresponse Adjustment 
4.1 Justification 


Generally in a survey, the base weights are computed from the inverse of the probability of selection for 
the sample. However, since the Program for International Student Assessment Young Adult Follow-up 
Study (PISA YAFS) is a follow-up to PISA 2012, the base weights for PISA YAFS are the final PISA 
2012 student weights. These PISA 2012 student weights were then adjusted to compensate for PISA 
YAFS nonresponse. There were two sources of nonresponse—those PISA students that did not provide 
follow-up contact information in 2012 during the PISA assessment, and those that did not respond to the 
PISA YAFS survey. 


4.2 Response Propensity Scores 


Nonresponse adjustment (NRA) to the PISA 2012 student weights was conducted to create new weights 
for the PISA YAFS data. The NRA was a two-step process that first adjusted for those participants who 
did not provide contact information after the 2012 assessment, then adjusted for the participants who did 


not respond to the Education and Skills Online assessment (ESO). 


The weighting classes for the NRA are created through a two-step procedure to group participants with 
similar characteristics related to nonresponse. The first step is the variable selection, which selects a 
subset of variables for use in model construction. The second step employs a regression analysis to 
estimate response propensities for creating the weighting classes. The weights are then adjusted within the 


weighting classes as described in the next section. 


First, the variable selection step was performed through Least Absolute Shrinkage and Selection Operator 
(LASSO) regression, which is a penalized or regularized regression from the field of machine learning 
(Tibshirani 1996). The LASSO regression shrinks nonsignificant regression coefficient estimates to zero 
and produces a simpler model that includes only a subset of the predictors. A tuning parameter controls 
the strength of the penalty that helps determine the number of remaining predictors. LASSO solutions are 
quadratic programming problems that are solved the same as minimizing the sum of squares with 
constraints on the summation of the coefficients. The main assumption in the LASSO regression is that 


response status has a sparse model and can be explained by a small subset of predictors. Only the 


PISA YAFS Technical Report and User Guide 21 


variables most predictive of PISA YAFS nonresponse identified by LASSO are retained for the 


subsequent step. 


PISA YAFS is unusual in that there are a large number of possible predictor variables available from the 
PISA 2012 background questionnaire. There were over 450 PISA background variables entered into 
LASSO that identified 53 variables for response propensity modeling. The bias increases as the penalty 
increases and more coefficients are set to zero. A propensity score analysis was run that showed a 
reasonable overlap in the distributions for the responding and nonresponding groups that indicated the 
groups have common support that helped determine the number of variables kept in the final model. 
Response propensity scores were created using this subset of variables. Logistic regression was performed 


to get an estimate of the response propensity for each case and to create weighting classes. 


4.3 Nonresponse Adjusted Weights 


The propensity response was sorted and 20 equal-sized NRA weighting classes were created. The 


nonresponse adjustment factors were then calculated and NRA weights created in each weighting class. 


The weight adjustments between the nonresponse cells varied. For one group in particular, the weighted 
response rate was very low, at 5.57 percent, leading to a high NRA factor of 17.97. The maximum value 
of PISA YAFS student weight (W_YFSTUWT) is 16181.9, compared to the maximum value of PISA 
student weight (W_FSTUWT of 2597.9). It was decided the weights would be left as is, without trimming 
the highest weights or collapsing the lowest nonresponse cells. This decision was made based on a series 
of factors. The increase in the weight variation, as measured by the Coefficient of Variation, was not very 
large, increasing from 41.22 to 75.61. Thus, the design effect increased from 1.17 to 1.57 which was 
deemed reasonable given the overall response rate and the size of the adjustment factors. Further, 
characteristics of PISA respondents across the 20 nonresponse cells were examined to see whether there 
was evidence that the nonresponse cells were related to educational attainment. Across many variables, 
but in particular the plausible values for reading, math, and science, it was clear that proficiency increases 
as propensity to respond increases across the nonresponse cell groups. This indicates a large nonresponse 


bias, substantially reduced by the NRA. 
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5. Nonresponse Bias Analysis 
5.1 Introduction 


The PISA YAFS total eligible sample comprised young adults who, as students, participated in the PISA 
2012 assessment and did not take the PISA 2012 financial literacy component. This resulted in a total 
eligible sample of 4,978 young adults, as they were all determined to be eligible. Of these, 2,318 
participated (the participating sample) for an unweighted response rate of 46.6 percent (45.9 percent 
weighted). 


NCES standards for assessment surveys stipulate that a nonresponse bias analysis is required at any stage 
of data collection reporting a weighted unit response rate of less than 85 percent. Since the PISA YAFS 
weighted response rate is below 85 percent, NCES standards require an investigation into the potential 


magnitude of nonresponse bias in the sample. 


5.2 Methodology 


To measure the potential nonresponse bias, the characteristics of participating young adults were 
compared to those of the total eligible sample of young adults. This was conducted in a way so that the 
tests of statistical significance that were applied account for the fact that the participating young adults are 


a subset of the eligible young adults, and not a distinct group. 


The general approach taken involves an analysis in two parts as described below: 


7 Analysis of the participating sample: the distribution of the participating sample 
(N = 2,318) was compared with that of the total eligible sample (N = 4,978). In each 
sample, young adults were weighted by their base weights, excluding any PISA YAFS 
nonresponse adjustment factor. The base weight for each young adult is the final PISA 
2012 student weight that includes both the PISA school and student nonresponse 
adjustments (NRAs). 


7 Analysis of the NRA sample: the same sets of young adults were compared as in the 
first analysis but this time, when analyzing the participating young adults, the PISA 
YAFS NRAs were applied to the weights. The total eligible sample was again 
weighted by their base weights, excluding any PISA YAFS NRA factor. 
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The first analysis indicates the potential for nonresponse bias that was introduced through young adult 
nonresponse. The second analysis indicates the potential for bias after accounting for the mitigating 


effects of nonresponse weight adjustments. 


Participating young adults and the total eligible young adult sample were compared by student responses 
to selection questions from the PISA 2012 background questionnaire that might provide information 
about the presence of nonresponse bias. Comparing these responses between participating young adults 
and the total eligible sample is not an ideal measure of nonresponse bias if the characteristics are 
unrelated or weakly related to more substantive items in the survey; however, it is the best approach 


available since PISA YAFS data are not available for nonparticipating young adults. 


There are a large number of variables available from the PISA 2012 student background questionnaire. To 
select the variables for this analysis, we started with the variables that were used in the nonresponse 
adjustment described in chapter 4. This was because variables had to be highly related to participation for 
inclusion in the final weighting model. We included all the weighting variables in a preliminary analysis, 
and the results were the same for all the variables in that they were highly significant for the participating 
sample in part one of the analysis and not significant with nonresponse-adjusted weights applied in the 
second part. These results confirmed that the nonresponse adjustment worked as expected and met the 
objective of reducing the bias in all the weighting variables. Thus, rather than showing all 52 variables 
with the same result, we only show the results for 6 categorical variables that had a p value of 0.1 or less 
in the second analysis that is representative of all weighting variables, plus the key demographic 
variables: gender and race/ethnicity. Additionally, a few key variables used in the YAFS analysis but not 
used in weighting were also included. All the variables are given in appendix A. The variables used in the 


YAFS analysis but not used in weighting are identified with an (A) in the following lists. 


The following categorical variables were available for all young adults: 


7 Derived race/ethnicity—missing, White, Black or African American, Hispanic, Asian, 
Two or more races, and other; 


7 Gender—female and male; 


a Attended preschool or kindergarten—missing, no, yes, for 1 year or less, and yes, for 
more than one year; 


L Attended preschool—missing, no, yes, for one year or less, and yes, for more than | 
year; 
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5 Truancy - skipped whole school day—amissing, none, one or two times, three or four 
times, and five or more times; 


. Highest grade expected to complete— 
2 missing, 
S less than high school, 
= high school (HS diploma or GED), 
7 vocational, or technical certificate, 
- associate’s degree (2-year college degree), 
7 bachelor’s degree (4-year college degree), 
2 master’s degree, and 
- doctoral or professional degree; 


o Highest level of parental educational attainment completed (classified into 
International Standard Classification of Education, or ISCED level) (A)— 


= missing, 

- none, 

= grade 6, 

- grade 9, 

= high school diploma, GED, vocational, or technical certificate/diploma, 
- associate’s degree (2-year college degree), and 


= bachelor’s degree (4-year college degree), master’s degree, and doctoral or 
professional degree; 


a Father’s current job status (A)—missing, working full-time for pay, working part-time 
for pay, not working, but looking for a job, and other (e.g., home duties, retired); 


7 Language spoken at home most of the time (A)—missing, Spanish, English, and 
another language; and 


7 Sense of belonging - feel happy at school (A)—missing, strongly agree, agree, 
disagree, strongly disagree. 


PISA YAFS Technical Report and User Guide 25 


The following continuous variables were available for all young adults: 


7 The first plausible value in mathematics, reading, and science; and 


. Openness for problem solving (A). 


For categorical variables, the distribution of frame characteristics for participants was compared with the 
distribution for all eligible young adults. The hypothesis of independence between the characteristic and 
participation status was tested using a Rao-Scott modified Chi-square statistic at the 5 percent level (Rao 
and Thomas 2003). For continuous variables, summary means were calculated and the difference between 
means was tested using a ¢ test. The p values for the tests are presented in the tables that follow. The 
statistical significance of differences between participants and the total eligible sample is identical to that 
which would result from comparing participants and nonparticipants, since all significance tests account 
for the participants being a subset of the full sample. The bias and relative bias are also shown in each 
table. The bias is calculated as the difference between the respective estimates for the participants and the 
eligible sample. The relative bias is calculated as the bias divided by the estimate from the eligible 


sample. The relative bias is a measure of the size of the bias compared to the eligible sample estimate. 


In addition to these tests, logistic regression models were used to provide a multivariate analysis that 
examined the conditional independence of these characteristics as predictors of participation. It may be 
that only one or two variables are actually related to participation status. However, if these variables are 
also related to the other variables examined in the analyses, then other variables, which are not related to 
participation status, will appear as significant in simple bivariate tables. Dummy variables were created 
for each component of the categorical variables so that each component was included separately. The last 
component of each categorical variable is used as the reference category. The p value of a dummy 
variable indicates whether there is a significant difference at the 5 percent level from the effect of the 
(omitted) reference category. Only the variables that had a p value of 0.1 or less in the second analysis 
were included in the regression. Gender, derived race/ethnicity, and the plausible value variables were not 
included in the regression model for this reason. Openness for problem solving was also not included in 
the regression due to the large number (1,799) of observations with missing values that would be dropped 
from the regression model. The concept of NRA weights does not apply to the nonresponding units; thus, 
a regression analysis that compares respondents with nonrespondents using NRA weights could not 


conducted in the second part of the analysis. 
The analysis was performed using WesVar® (Westat 2007) and replicate weights to properly account for 


the complex sample design. The Fay method of balanced repeated replication (BRR) was used to create 


the replicate weights (Westat 2007). 
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5.3 Participating Sample 


This section presents the nonresponse bias analysis based on the sample of 4,978 eligible young adults for 
the PISA YAFS sample. The distribution of the participating sample was compared to the young adults in 
the total eligible sample. Base weights were used for both the eligible sample and the participating young 

adults. The unweighted response rate for PISA YAFS was 46.6 percent, with 2,318 out of 4,978 young 


adults participating. The weighted response rate was 45.9 percent. 


5.3.1 Categorical Variables 


The distribution of participating and eligible young adults by the characteristics are shown in tables 5, 6, 
and 7. The Chi-square statistic for derived race/ethnicity, gender, attended preschool or kindergarten, 
attended preschool, truancy - skipped whole school day, highest grade expected to complete, highest level 
of parental educational attainment, and father’s current job status was significant, suggesting evidence of 
relationships with participation in the assessment. Significant differences between eligible young adults 


and those that participated are as follows: 


a Black or African American young adults were underrepresented among participating 
young adults relative to eligible young adults (9.1 versus 12.5 percent, respectively), 
while White young adults were overrepresented among participating young adults 
(53.8 versus 50.4 percent, respectively), table 5. 


7 Male young adults were underrepresented among participating young adults relative 
to eligible young adults (44.7 versus 51.0 percent, respectively), while female young 
adults were overrepresented among participating young adults (55.3 versus 49.0 
percent, respectively), table 5. 


7 Young adults who did not attend preschool or kindergarten were underrepresented 
among participating young adults relative to eligible young adults (1.2 versus 1.5 
percent, respectively), while young adults who attended preschool or kindergarten for 
more than | year were overrepresented among participating young adults (74.8 versus 
73.7 percent, respectively), table 5. 


m1 Young adults who attended preschool for | year or less were underrepresented among 
participating young adults relative to eligible young adults (43.4 versus 45.7 percent, 
respectively), while young adults who attended for more than 1 year were 
overrepresented among participating young adults (31.5 versus 27.5 percent, 
respectively), table 5. 
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5 Young adults who were truant one or two times were underrepresented among 
participating young adults relative to eligible young adults (16.5 versus 17.7 percent, 
respectively), while young adults who were never truant were overrepresented among 
participating young adults (81.0 versus 78.1 percent, respectively), table 6. 


7 Young adults who expected to complete high school were underrepresented among 
participating young adults relative to eligible young adults (4.9 versus 7.7 percent, 
respectively), while young adults who expected to complete a doctoral or professional 
degree were overrepresented among participating young adults (27.0 versus 22.9 
percent, respectively), table 6. 


7 Young adults with the highest level of parental educational attainment of High school, 
GED, vocational, or technical certificate/diploma were underrepresented among 
participating young adults relative to eligible young adults (28.2 versus 31.7 percent, 
respectively), while young adults with Bachelor’s, master’s, and doctoral or 
professional degree were overrepresented among participating young adults (48.9 
versus 43.2 percent, respectively), table 7. 


7 Young adults whose father’s current job status was working part-time with pay were 
underrepresented among participating young adults relative to eligible young adults 
(5.7 versus 6.4 percent, respectively), while young adults whose father was working 
full-time for pay were overrepresented among participating young adults (76.0 versus 
72.8 percent, respectively, table 7. 


There were no statistically significant relationships between participation status and language spoken at 


home most of the time or sense of belonging — feel happy at school (table 7). 
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Table 5. Percentage distribution of eligible and participating young adults in the PISA YAFS 
participating sample, by derived race/ethnicity, gender, attended preschool or kindergarten, and 
attended preschool: 2016 


Sample 
Eligible _— Participating 
(percent) (percent) Relative Chi-square 
Characteristic (N = 4,978) (N = 2,318) Bias bias p value 
Derived race/ethnicity 0.000 
Missing 1.4 0.6 -0.80 -0.571 
White 50.4 53.8 3.40 0.067 
Black or African 
American 12.5 9.1 -3.40 -0.272 
Hispanic 24.5 24.0 -0.50 -0.020 
Asian 4.7 6.7 2.00 0.426 
Two or more races 4.6 4.3 -0.30 -0.065 
Other 2.0 1.5 -0.50 -0.250 
Gender 0.000 
Female 49.0 55.3 6.30 0.129 
Male 51.0 44.7 -6.30 -0.124 
Attended preschool or 
kindergarten 0.000 
Missing 1.2 0.3 -0.90 -0.750 
No 1.5 1.2 -0.30 -0.200 
Yes, for 1 year or 
less 23.7 23.8 0.10 0.004 
Yes, for more than 
1 year 73.7 74.8 1.10 0.015 
Attended preschool 0.000 
Missing 1.8 0.8 -1.00 -0.556 
No 25.0 24.4 -0.60 -0.024 
Yes, for 1 year or 
less 45.7 43.4 -2.30 -0.050 
Yes, for more than 
1 year 27.5 31.5 4.00 0.145 


NOTE: Detail may not sum to totals because of rounding. The bias is calculated as the difference between the respective 
estimates for the participating and eligible sample. The relative bias is calculated as the bias divided by the estimate from the 
eligible sample. Young adults were weighted by their PISA YAFS base weights that did not include a PISA YAFS nonresponse 
adjustment factor. Bolded p values indicate statistical significance at the 5 percent level. 

SOURCE: U.S. Department of Education, National Center for Education Statistics, PISA Young Adult Follow-up Study (PISA 
YAFS), 2016. 


PISA YAFS Technical Report and User Guide 29 


Table 6. Percentage distribution of eligible and participating young adults in the PISA YAFS 


participating sample, by truancy and highest grade expected to complete: 2016 


Sample 
Eligible —_ Participating 
(percent) (percent) 
Characteristic (N = 4,978) (N = 2,318) 
Truancy — Skipped 
whole school day 
Missing 1.1 0.2 
None 78.1 81.0 
One or two times 17.7 16.5 
Three or four times 2.4 1.9 
Five or more times 0.8 0.4 
Highest grade expected to 
complete 
Missing 1.7 0.6 
Less than high 
school 1.5 0.8 
High school (HS 
diploma or 
GED) 7.7 4.9 
Vocational or 
technical 
certificate 3.8 25 
Associate’s degree 6.1 4.7 
Bachelor’s degree 34.0 35.1 
Master’s degree 22.3 24.3 
Doctoral or 
professional 
degree 22.9 27.0 


Bias 


-0.90 

2.90 
-1.20 
-0.50 
-0.40 


-1.10 


-0.70 


-2.80 


-1.30 
-1.40 
1.10 
2.00 


4.10 


Relative 
bias 


-0.818 

0.037 
-0.068 
-0.208 
-0.500 


-0.647 


-0.467 


-0.364 


-0.342 
-0.230 
0.032 
0.090 


0.179 


Chi-square 
p value 


0.000 


0.000 


NOTE: Detail may not sum to totals because of rounding. The bias is calculated as the difference between the respective 
estimates for the participating and eligible sample. The relative bias is calculated as the bias divided by the estimate from the 
eligible sample. Young adults were weighted by their PISA YAFS base weights that did not include a PISA YAFS nonresponse 


adjustment factor. Bolded p values indicate statistical significance at the 5 percent level. 


SOURCE: U.S. Department of Education, National Center for Education Statistics, PISA Young Adult Follow-up Study (PISA 


YAFS), 2016. 
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Table 7. Percentage distribution of eligible and participating young adults in the PISA YAFS 
participating sample, by highest level of parental educational attainment, father’s current job 
status, language spoken at home, and sense of belonging: 2016 


Sample 
Eligible Participating 
(percent) (percent) Relative Chi-square 
Characteristic (N =4,978) (N=2,318) Bias bias p value 
Highest level of parental 
educational attainment 0.000 
Missing 2.2 1.0 -1.20 -0.545 
None 0.9 1.1 0.20 0.222 
Grade 6 2.6 2.3 -0.30 -0.115 
Grade 9 5.4 4.4 -1.00 -0.185 
High school, GED, vocational, or 
technical certificate/diploma 31.7 28.2 -3.50 -0.110 
Associate’s degree 14.1 14.0 -0.10 -0.007 
Bachelor’s, master’s, and doctoral 
or professional degree 43.2 48.9 5.70 0.132 
Father current job status 0.000 
Missing 7.8 5.9 -1.90 -0.244 
Working full-time for pay 72.8 76.0 3.20 0.044 
Working part-time for pay 6.4 ae! -0.70 -0.109 
Not working, but looking for a job 4.4 4.2 -0.20 -0.045 
Other (e.g., home duties, retired) 8.6 8.2 -0.40 -0.047 
Language spoken at home most of 
the time 0.078 
Missing 2.2 0.8 -1.40 -0.636 
Spanish 10.7 9.6 -1.10 -0.103 
English 83.7 85.8 2.10 0.025 
Another language 3.4 3.8 0.40 0.118 
Sense of belonging - feel happy at 
school 0.075 
Missing 35.6 252 -0.10 -0.003 
Strongly agree 13.6 14.4 0.80 0.059 
Agree yee 37.8 0.10 0.003 
Disagree 11.0 10.8 -0.20 -0.018 
Strongly disagree 2.1 1.5 -0.60 -0.286 


NOTE: Detail may not sum to totals because of rounding. The bias is calculated as the difference between the respective 
estimates for the participating and eligible sample. The relative bias is calculated as the bias divided by the estimate from the 
eligible sample. Young adults were weighted by their PISA YAFS base weights that did not include a PISA YAFS nonresponse 
adjustment factor. Bolded p values indicate statistical significance at the 5 percent level. 

SOURCE: U.S. Department of Education, National Center for Education Statistics, PISA Young Adult Follow-up Study (PISA 
YAFS), 2016. 
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5.3.2 Continuous Variables 


Summary means for each continuous variable for participating and eligible young adults are shown in 
tables 8 and 9. The ¢ test statistic for all continuous variables in this section were significant, suggesting 


evidence of relationships with participation in the assessment. 


Participating young adults had a higher mean of plausible value | in mathematics, reading, and science 
than the eligible sample (507.6 versus 480.7, 525.2 versus 497.0, and 524.8 versus 497.2, respectively; 


table 8) and a higher mean of openness for problem solving than the eligible sample (0.3 versus 0.2; 


Table 8. Mean scores of plausible value | for eligible and participating young adults in the PISA YAFS 
participating sample: 2016 


Sample 
Eligible Participating 

(mean score) (mean score) Relative t test 
Characteristic (N = 4,978) (N = 2,318) Bias bias p value 

Plausible value | in 
Mathematics 480.7 507.6 26.83 0.056 0.000 
Reading 497.0 525.2 28.22 0.057 0.000 
Science 497.2 524.8 27.61 0.056 0.000 


NOTE: Mathematics, reading, and science scores are based on original student PISA 2012 scores. The bias is calculated as the 
difference between the respective estimates for the participating and eligible sample. The relative bias is calculated as the bias 
divided by the estimate from the eligible sample. Young adults were weighted by their PISA YAFS base weights that did not 
include PISA YAFS nonresponse adjustment factor. Bolded p values indicate statistical significance at the 5 percent level. 
SOURCE: U.S. Department of Education, National Center for Education Statistics, PISA Young Adult Follow-up Study (PISA 
YAFS), 2016. 


Table 9. Mean values of openness for problem solving for eligible and participating young adults in the 
PISA YAFS participating sample: 2016 


Sample young adults 
Eligible Participating 


(mean) (mean) Relative t test 

Characteristic (N = 3,179) (N = 1,515) Bias bias p value 
Openness for problem 

solving 0.2 0.3 0.11 0.611 0.000 


NOTE: The bias is calculated as the difference between the respective estimates for the participating and eligible sample. The 
relative bias is calculated as the bias divided by the estimate from the eligible sample. Young adults were weighted by their PISA 
YAFS base weights that did not include a PISA YAFS nonresponse adjustment factor. Bolded p values indicate statistical 
significance at the 5 percent level. 

SOURCE: U.S. Department of Education, National Center for Education Statistics, PISA Young Adult Follow-up Study (PISA 
YAFS), 2016. 
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5.3.3 Logistic Regression Model 


To examine the joint relationship of various characteristics to nonresponse, the analysis used a logistic 
regression model with participation status as the binary dependent variable and frame characteristics as 
predictor variables. To limit the number of predictor variables in the model, only those weighting 
variables with a p value less than 0.1 with nonresponse-adjusted weights in section 5.4 and the categorical 


analysis variables were included. 


Standard errors and tests of hypotheses for the full model parameter estimates are presented in table 10. 
Eleven characteristics and the interaction term were significant predictors of school participation. The 


positive parameter estimates indicate that: 


a young adults with missing data were somewhat overrepresented among participating 
young adults relative to young adults who reported a low sense of belonging (sense of 
belonging - feel happy at school = strongly disagree, table 10). 


The negative parameter estimates indicate that 


7 young adults who attended preschool | year or less were somewhat underrepresented 
among participating young adults relative to young adults who attended preschool for 
more than 1 year; 


7 young adults with other educational expectations (highest grade expected to complete 
being missing, less than high school, high school [HS diploma or GED], vocational or 
technical certificate, associate’s degree [2-year college degree], and bachelor’s degree 
[4-year college degree]), were somewhat underrepresented among participating young 
adults relative to participating young adults who expected to complete a doctoral or 
professional degree; 


Ey young adults whose parents’ highest completed level of education was High school, 
GED, vocational, or technical certificate/diploma were somewhat underrepresented 
among participating young adults relative to young adults whose parents’ highest 
completed level of education was bachelor’s, master’s, and doctoral or professional 
degree; and 


7 young adults with missing data were somewhat underrepresented among participating 


young adults relative to young adults who spoke another language at home other than 
English or Spanish, table 10. 
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Table 10. Logistic regression model parameters using the PISA YAFS young adult sample: 2016 


Parameter Standard t test for Ho: 
Parameter estimate error parameter = 0 p value 
Intercept -0.68 0.523 -1.310 0.194 
Attended pre-school or kindergarten 
missing -0.46 0.528 -0.867 0.388 
No 0.02 0.371 0.043 0.966 
Yes, for 1 year or less 0.45 0.282 1.587 0.116 
Attended pre-school 
missing -0.54 0.460 -1.171 0.245 
No -0.54 0.288 -1.863 0.066 
Yes, for 1 year or less -0.28 0.082 -3.446 0.001 
Truancy - Skipped whole school day 
missing 0.23 0.722 0.314 0.754 
None 0.87 0.478 1.812 0.074 
One or two times 0.79 0.490 1.606 0.112 
Three or four times 0.58 0.505 1.154 0.252 
Highest grade expected to complete 
missing -0.92 0.395 -2.335 0.022 
Less than high school -1.05 0.292 -3.589 0.001 
High school (HS diploma or GED) -0.86 0.142 -6.082 0.000 
Vocational or technical certificate -0.87 0.199 -4.362 0.000 
Associate’s degree -0.65 0.127 -5.123 0.000 
Bachelor’s degree -0.25 0.070 -3.610 0.001 
Master’s degree -0.16 0.092 -1.779 0.079 
Highest level of parental educational attainment 
missing -0.43 0.271 -1.574 0.119 
None 0.63 0.323 1.953 0.054 
Grade 6 -0.11 0.261 -0.415 0.679 
Grade 9 -0.28 0.164 -1.688 0.095 
High school, GED, vocational, or technical 
certificate/diploma -0.27 0.092 -2.939 0.004 
Associate’s degree -0.17 0.124 -1.355 0.179 
Father current job status 
missing -0.11 0.165 -0.684 0.496 
Working full-time for pay 0.10 0.096 1.038 0.302 
Working part-time for pay -0.07 0.151 -0.434 0.666 
Not working, but looking for a job 0.08 0.161 0.474 0.637 
Language spoken at home most of the time 
Missing -1.15 0.364 -3.155 0.002 
Spanish -0.33 0.208 -1.603 0.113 
English -0.21 0.178 -1.166 0.247 
Sense of belonging - Feel happy at school 
missing 0.45 0.223 2.004 0.048 
Strongly agree 0.49 0.225 2.173 0.033 
Agree 0.41 0.223 1.853 0.067 
Disagree 0.43 0.262 1.641 0.105 


| 
NOTE: Young adults were weighted by their PISA YAFS base weights that did not include PISA YAFS nonresponse adjustment 
factor. Bolded p values indicate statistical significance at the 5 percent level. 

SOURCE: U.S. Department of Education, National Center for Education Statistics, PISA Young Adult Follow-up Study (PISA 
YAFS), 2016. 
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5.4 Nonresponse Adjusted Sample 


This section presents the nonresponse bias analysis based on the NRA sample of 4,978 eligible young 
adults for the PISA YAFS sample. The distribution of the participating sample was compared to the 
young adults in the total eligible final sample, just like the previous section. However, in the analyses that 
follow, base weights were used for the eligible sample of young adults, whereas PISA YAFS NRA 


weights were used for the participating young adults. 


5.4.1 Categorical Variables 


The distribution of participating and eligible young adults by the characteristics is shown in tables 11, 12, 
and 13. Only the Chi-square statistic for “attended preschool” and “highest grade expected to complete” 
remained significant once nonresponse adjusted weights were applied, which suggests that there is some 


evidence of relationships with participation in the assessment. In particular: 


P Young adults who attended preschool for 1 year or less were overrepresented among 
participating young adults relative to eligible young adults (46.6 versus 45.7 percent, 
respectively), while young adults who attended for more than 1 year were 
underrepresented among participating young adults (26.6 versus 27.5 percent, 
respectively), table 11. 


: Young adults who expected to complete high school as the highest level of 
educational attainment were overrepresented among participating young adults 
relative to eligible young adults (8.7 versus 7.7 percent, respectively), while young 
adults who expected to complete a master’s degree were underrepresented among 
participating young adults (21.6 versus 22.3 percent, respectively), table 12. 


For both of these characteristics, the NRA slightly over-adjusted for most of the categories (as seen by the 
change in the sign of the bias) but still greatly reduced the bias in all categories of “highest grade 
expected to complete” (table 12) and all but one in “attended preschool” (table 11). 
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Table 11. Percentage distribution of eligible and participating young adults in the PISA YAFS 
nonresponse-adjusted sample, by derived race/ethnicity, gender, attended preschool or 
kindergarten, and attended preschool: 2016 


Sample 
Eligible —_ Participating 
(percent) (percent) Relative Chi-square 
Characteristic (N = 4,978) (N = 2,318) Bias bias p value 
Derived race/ethnicity 0.107 
Missing 1.4 0.6 -0.80 -0.571 
White 50.4 51.1 0.70 0.014 
Black or African 
American 12.5 12.6 0.10 0.008 
Hispanic 24.5 24.7 0.20 0.008 
Asian 4.7 5.0 0.30 0.064 
Two or more 
races 4.6 4.1 -0.50 -0.109 
Other 2.0 2.0 0.00 0.000 
Gender 0.955 
Female 49.0 48.9 -0.10 -0.002 
Male 51.0 51.1 0.10 0.002 
Attended preschool or 
kindergarten 0.054 
Missing 1.2 0.5 -0.70 -0.583 
No 1.5 1.4 -0.10 -0.067 
Yes, for 1 year or 
less 23.7 24.2 0.50 0.021 
Yes, for more 
than | year 73.7 73.9 0.20 0.003 
Attended preschool 0.029 
Missing 1.8 1.0 -0.80 -0.444 
No 25.0 25.8 0.80 0.032 
Yes, for 1 year or 
less 45.7 46.6 0.90 0.020 
Yes, for more 
than | year 27.5 26.6 -0.90 -0.033 


NOTE: Detail may not sum to totals because of rounding. The bias is calculated as the difference between the respective 
estimates for the participating and eligible sample. The relative bias is calculated as the bias divided by the estimate from the 
eligible sample. Participating young adults were weighted by their PISA YAFS nonresponse adjusted weight. Eligible young 
adults were weighted by their PISA YAFS base weights. Bolded p values indicate statistical significance at the 5 percent level. 
SOURCE: U.S. Department of Education, National Center for Education Statistics, PISA Young Adult Follow-up Study (PISA 
YAFS), 2016. 
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Table 12. Percentage distribution of eligible and participating young adults in the PISA YAFS 
nonresponse-adjusted sample, by truancy and highest grade expected to complete: 2016 


Characteristic 


Truancy — Skipped 
whole school day 
Missing 
None 
One or two times 
Three or four times 
Five or more times 


Highest grade expected 
to complete 
Missing 
Less than high 
school 
High school (HS 
diploma or GED) 
Vocational or 
technical 
certificate 
Associate’s degree 
Bachelor’s degree 
Master’s degree 
Doctoral or 
professional 
degree 


Sample 
Eligible Participating 
(percent) (percent) 
(N=4,978) (N= 2,318) 
1.1 0.3 
78.1 77.6 
17.7 18.6 
2.4 25 
0.8 1.0 
1.7 0.8 
1.5 1.2 
Td. 8.7 
3.8 4.6 
6.1 6.0 
34.0 g3.5 
22.3 21.6 
22.9 23.6 


Bias 


-0.80 
-0.50 
0.90 
0.10 
0.20 


-0.90 
-0.30 
1.00 
0.80 
-0.10 


-0.50 
-0.70 


0.70 


Relative 
bias 


-0.727 
-0.006 
0.051 
0.042 
0.250 


-0.529 
-0.200 
0.130 
0.211 
-0.016 


-0.015 
-0.031 


0.031 


Chi-square 
p value 


0.076 


0.034 


NOTE: Detail may not sum to totals because of rounding. The bias is calculated as the difference between the respective 
estimates for the participating and eligible sample. The relative bias is calculated as the bias divided by the estimate from the 
eligible sample. Participating young adults were weighted by their PISA YAFS nonresponse adjusted weight. Eligible young 
adults were weighted by their PISA YAFS base weights. Bolded p values indicate statistical significance at the 5 percent level. 
SOURCE: U.S. Department of Education, National Center for Education Statistics, PISA Young Adult Follow-up Study (PISA 


YAFS), 2016. 
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Table 13. Percentage distribution of eligible and participating young adults in the PISA YAFS 
nonresponse-adjusted sample, by highest level of parental educational attainment, father’s 
current job status, language spoken at home, and sense of belonging: 2016 


Sample 
Eligible Participating 
(percent) (percent) 
Characteristic (N=4,978) (N=2,318) 
Highest level of parental 
educational attainment 
Missing 22 1.4 
None 0.9 0.9 
Grade 6 2.6 3.0 
Grade 9 5.4 5.6 
High school, GED, vocational, or 
technical certificate/diploma 31.7 31.1 
Associate’s degree 14.1 15.1 
Bachelor’s, master’s, 
and doctoral or professional 
degree 43.2 42.9 
Father current job status 
Missing 7.8 8.8 
Working full-time for pay 72.8 72.1 
Working part-time for pay 6.4 5.5 
Not working, but looking for a 
job 4.4 4.8 
Other (e.g., home duties, retired) 8.6 8.8 
Language at home 
Missing 2.2 22 
Spanish 10.7 10.6 
English 83.7 83.4 
Another language 3.4 3.8 
Sense of belonging — feel 
happy at school 
Missing 35.6 36.5 
Strongly agree 13.6 14.4 
Agree 37.7 36.5 
Disagree 11.0 11.1 
Strongly disagree 2.1 1.4 


Bias 


-0.80 
0.00 
0.40 
0.20 


-0.60 
1.00 


-0.30 


1.00 
-0.70 
-0.90 


0.40 
0.20 


0.00 
-0.10 
-0.30 

0.40 


0.90 
0.80 
-1.20 
0.10 
-0.70 


Relative 
bias 


-0.364 
0.000 
0.154 
0.037 


-0.019 
0.071 


-0.007 


0.128 
-0.010 
-0.141 


0.091 
0.023 


0.000 
-0.009 
-0.004 

0.118 


0.025 
0.059 
-0.032 
0.009 
-0.333 


Chi-square 
p value 


0.217 


0.224 


0.704 


0.111 


NOTE: Detail may not sum to totals because of rounding. The bias is calculated as the difference between the respective 
estimates for the participating and eligible sample. The relative bias is calculated as the bias divided by the estimate from the 
eligible sample. Participating young adults were weighted by their PISA YAFS nonresponse adjusted weight. Eligible young 
adults were weighted by their PISA YAFS base weights. Bolded p values indicate statistical significance at the 5 percent level. 
SOURCE: U.S. Department of Education, National Center for Education Statistics, PISA Young Adult Follow-up Study (PISA 


YAFS), 2016. 
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5.4.2 Continuous Variables 


Summary means for each continuous variable for participating and eligible young adults are shown in 
tables 14 and 15. There were no statistically significant differences between participating and eligible 


young adults across any of the characteristics shown in tables 14 and 15 once NRA weights were applied. 


Table 14. Mean scores of plausible values for eligible and participating young adults in the PISA YAFS 
nonresponse-adjusted sample: 2016 


Sample 
Eligible Participating 

(mean score) (mean score) Relative t test 
Characteristic (N = 4,978) (N = 2,318) Bias bias p value 

Plausible value 1 in 
Mathematics 480.72 481.70 0.98 0.002 0.560 
Reading 497.02 498.65 1.63 0.003 0.416 
Science 497.19 498.38 1.19 0.002 0.501 


NOTE: Mathematics, reading, and science scores are based on original student PISA 2012 scores. The bias is calculated as the 
difference between the respective estimates for the participating and eligible sample. The relative bias is calculated as the bias 
divided by the estimate from the eligible sample. Participating young adults were weighted by their PISA YAFS nonresponse 
adjusted weight. Eligible young adults were weighted by their PISA YAFS base weights. Bolded p values indicate statistical 
significance at the 5 percent level. 

SOURCE: U.S. Department of Education, National Center for Education Statistics, PISA Young Adult Follow-up Study (PISA 
YAFS), 2016. 


Table 15. Mean values of openness for problem solving for eligible and participating young adults in 
the YAFS nonresponse-adjusted sample: 2016 


Sample 
Eligible Participating 
(mean) (mean) Relative t test 
Characteristic (N = 3,179) (N = 1,515) Bias bias p value 
Openness for problem 
solving 0.18 0.16 -0.02 -0.111 0.474 


NOTE: The bias is calculated as the difference between the respective estimates for the participating and eligible sample. The 
relative bias is calculated as the bias divided by the estimate from the eligible sample. Participating young adults were weighted 
by their PISA YAFS nonresponse adjusted weight. Eligible young adults were weighted by their PISA YAFS base weights. 
Bolded p values indicate statistical significance at the 5 percent level. 

SOURCE: U.S. Department of Education, National Center for Education Statistics, PISA Young Adult Follow-up Study (PISA 
YAFS), 2016. 


5.5 Summary 


Since the PISA YAFS weighted response rate was below 85 percent, NCES standards require an 
investigation into the potential magnitude of nonresponse bias at the young adult level in the sample. The 


investigation into nonresponse bias for the PISA YAFS data collection effort showed statistically 


\o 
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significant relationships between response status and some of the available student characteristics that 


were examined in the analyses. 


For the analysis of the participating sample of young adults without NRAs applied to the weights, most 
variables were found to be statistically significantly related to participation in the bivariate analysis: 
derived race/ethnicity, gender, attended preschool or kindergarten, attended preschool, truancy - skipped 
whole school day, highest grade expected to complete, highest level of parental educational attainment, 
father’s current job status, plausible value 1 in mathematics, reading, and science, and openness for 
problem solving (table 5, table 6, table 7, table 8, and table 9). This was confirmed when all of these 
factors were considered simultaneously in a regression analysis; 11 of the parameter estimates were 


significant predictors of participation (table 10) important. 


For the participating sample of young adults with NRAs applied to the weights, only two variables 
remained statistically significantly related to participation in the bivariate analysis: attended preschool 
(table 11) and highest grade expected to complete (table 12). The multivariate regression analysis could 
not be conducted after the NRAs were applied to the weights. The concept of nonresponse-adjusted 
weights does not apply to the nonresponding units and thus, we could not conduct an analysis that 


compares respondents with nonrespondents using NRA weights. 


In sum, the investigation into nonresponse bias in the PISA YAFS sample provides evidence that there is 
limited nonresponse bias in the PISA YAFS participating sample based on the characteristics studied. 
This result is not surprising, and the bias is in the expected direction. For example, females responded at a 
higher level than males and higher achieving young adults responded at a higher level than other young 
adults did. However, after the application of NRAs, there is limited evidence of resulting potential bias in 
the sample. It is also important to note none of the variables used in the YAFS analysis but not used in 


weighting were significant after the application of NRAs. 


Overall, the nonresponse weighting reduced the bias on all the variables tested. The weighting did over- 
adjust somewhat on “attended preschool” and “highest grade expected to complete.” Since the 
nonresponse adjustment was based on many variables, not every variable can be controlled for in the 
adjustment. However, in every category for “highest grade expected to complete,” the absolute bias was 
reduced and by more than half for the majority, although the biases as a whole remained statistically 


significant. 
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6. Data Analysis and Scaling 


The systems of test administration and scoring employed for cognitive items were those in the ESO 
(OECD 2015), which are similar to the system used in PLAAC (OECD 2013). The analysis methods and 
procedures for PISA YAFS are based on identical psychometric principles as the methods and procedures 
used for the ESO and PIAAC (OECD 2013; 2015). 


Given the ESO assessment design described in chapter 3, each PISA YAFS respondent was administered 
a subset of ESO items from the total item pool. As a consequence of the design, different groups of 
respondents answered different sets of items, which makes it inappropriate to use any statistic based on 
the number of correct responses in reporting the survey results. Differences in total scores among 
respondents who took different sets of items may be due to variations in difficulty in the adaptively 
administered test forms. Unless one makes very strong and incorrect assumptions, such as perfectly 
parallel test forms, the performance of the different groups cannot be directly compared using total-score 
statistics. This limitation can be overcome by using Item Response Theory (IRT) scaling. Regularities in 
response patterns can be modeled using the underlying commonalities among the items and can be used to 
characterize respondents by estimating so-called person or ability parameters through IRT models. 
Response patterns can also be used to characterize items by estimating certain item parameters through 
IRT models; for example, item difficulty in terms of a common scale, even if not all respondents took 
identical sets of items. In other words, if an item pool is used to measure a certain unidimensional skill, 
respondents can be compared with each other even if they responded to different sets of items from this 
item pool. IRT makes it possible to describe distributions of performance in a population or 


subpopulation and to estimate the relationships between proficiency and background variables. 


Before data can be used for analyses, the quality of the data has to be evaluated. This evaluation was done 
by reviewing the item responses to determine whether each respondent received the items as planned in 
the design and performing quality checks to evaluate the handling and pattern of the missing values (i.e., 
missing by design, omitted by the respondent). During the scaling analyses, item fit statistics were 
evaluated to determine the extent to which item parameters obtained from the ESO field test (which was 
used to create the final ESO item parameters) work similarly for PISA YAFS (information on the ESO 


development can be found at https://www.oecd.org/skills/ESonline-assessment/). As a final step, 


plausible values (PVs) were generated — which are multiple imputations — drawn from a posterior 
distribution by combining the IRT scaling of the cognitive items with a latent regression model using 


information from the background questionnaire (BQ) in a population model. 
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In the following sections, more details about the data evaluation process, the scaling process, and the 
population model used for PISA YAFS are described. 


6.1 Data Handling and Data Quantity for ESO-modified Items 


The assurance of the data quality is an important step prior to the IRT scaling and population modeling. 
Reasonable and meaningful results can only be provided if the analyses are based on correct data. 
Procedures for the handling of missing data, data completion, and item analyses are illustrated below. All 
the items administered in PISA YAFS were automatically scored; thus, evaluation of scoring reliability 


was not needed. 


6.1.1 Handling of Missing Data 


Plausible values were estimated for the cognitive items; thus, this section discusses how the data from the 
core literacy, numeracy, and problem solving assessments were handled. The same scoring guidelines and 
procedures were followed as those applied in the ESO and PIAAC for the computer-based administration, 
except for data completion (see the section below). The literacy and numeracy items were dichotomously 
scored: correct responses were scored as 1, and incorrect responses as 0. For data analysis purposes, 
missing data were handled with a procedure similar to that used in ESO and PIAAC in order to maintain 
comparability among the studies. The structure of missing responses is mainly derived from the 


multistage adaptive testing design such as the following: 


5 Missing by design (scored as 9): Items that were not presented to each respondent due 
to the multistage adaptive design used in PISA YAFS. Accordingly, these structural 
missing data, unrelated to respondents’ literacy and numeracy skills, were ignored 
when calculating respondent proficiencies. 


7 Omitted responses (scored as 8): Missing responses that occurred when respondents 
chose not to perform one or more presented items, either because they were unable or 
for some other reason. Any missing response followed by a valid response (whether 
correct or incorrect) was defined as an omitted response. 


7 Not reached or not attempted responses (scored as 9): Missing responses at the end of 


a test were treated as if they were not presented due to the difficulty of determining if 
the respondent was unable to finish these items or simply abandoned them. 
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6.1.2 Data Quantity 


To ensure accurate proficiency estimations, the PISA YAFS sample respondents needed to provide valid, 
non-missing responses on both the BQ and cognitive items to be included in the item analysis described 
in section 6.1.4. A complete cognitive assessment, with a minimum of five attempted items per domain, 
was necessary to assure sufficient information about the proficiency of respondents. There were 2,318 
respondents in total who responded to at least three BQ items. Out of these respondents, 1,885 
respondents responded both to the BQ and completed at least one of the domains, either literacy or 
numeracy, or both domains. Thus, these 1,885 respondents served as the basis for the calculation of the 
standardized weights and evaluation of the item fits in the IRT scaling stage, and they received the 


plausible values in the population modeling stage. 


6.1.3 Data Completion - Treatment of Respondents With Fewer Than 5 
Cognitive Item Responses 


This section addresses the level of response required for a respondent to be eligible to receive PV scores. 
PISA YAFS followed the PIAAC Main Study procedure with respect to cases with responses to fewer 
than five cognitive items per domain. There were 1,790 respondents who provided sufficient background 
information and completed the cognitive assessment for both domains (literacy and numeracy). An 
additional 95 respondents (31 respondents completed all questions in the literacy domain, 62 respondents 
completed all questions in the numeracy domain, and 2 respondents completed both domains, but did not 
click to the final “thank you” screen) were considered to have enough data to be include in the analysis. In 
total, 1,790 respondents who completed both domains and 95 respondents who completed only one of the 
domains or did not click to the final screen to complete the test were considered eligible to receive the 
plausible values (PVs) (N = 1,885). 


6.1.4 Classical Test Theory Statistics: Item Analyses 
Once the data were prepared, item analyses were conducted separately for each domain (literacy and 
numeracy). The purpose of the item analyses was to identify outliers or unexpected patterns due to 


technical issues in the computer-based assessment data, such as data not following predictable patterns 


related to item difficulty or inconsistent response patterns. All descriptive statistics were examined for 
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observed responses as well as the various missing response codes. The following statistics for each item 
were computed and examined: 
. Item difficulties (proportion of correct responses, or P+); 


A frequencies of scores (number of respondents attempted, correct and incorrect 
responses, omitted items, not-reached items); and 


7 point biserial correlations. 
Additional details about the ESO item analyses procedures can be found in the international ESO 


technical documentation at https://www.oecd.org/skills/ESonline-assessment/assessmentdesign/ 
technicaldocumentation/ESOnline_ Technical Doc_Ch1l-8 Oct2016.pdf. 


6.2 IRT Scaling: Evaluation and Estimation of Item Parameters 


The ESO cognitive assessment consisted of 78 items: 40 items in the literacy domain and 38 items in the 
numeracy domain. The ESO computer delivery platform successfully delivered, captured, and exported 
information for these items. The IRT scaling provided estimations of item parameters and the preliminary 
proficiency distribution of the population. The IRT scaling was carried out separately for the domains of 
literacy and numeracy. Similar to ESO and PIAAC, PISA YAFS used the two-parameter logistic model 
(2PL) (Birnbaum 1968) for dichotomously scored responses. Incorrect responses were coded as 0, and 
correct responses were coded as 1 in the 2PL model; omitted responses were treated as incorrect 


responses, and missing by design responses and not reached items were treated as missing values. 


The 2PL model is a mathematical model for the probability that an individual will respond correctly to a 
particular item from a single domain of items. The probability of solving an item (i) depends only on the 
ability or proficiency (0j) of the respondent (j) and two item parameters characterizing the properties of 
the item (item difficulty B; and item discrimination a). This is related to the central assumption of 
conditional independence (sometimes also called local independence). That is, item response probabilities 
depend only on the respondent’s ability and the specified item parameters — there is no dependence on 
any demographic characteristics of the examinees or responses to any other items presented in a test or 
the survey administration conditions. Moreover, the 2PL model assumes unidimensionality, that is, a 
single latent variable, the ability or proficiency 8, accounts for an individual’s performance on a set of 
items. For more details about the models and IRT scaling process see the technical report for ESO (OECD 
2015) and the PIAAC main study (OECD 2013). 
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As PISA YAFS uses the ESO literacy and numeracy assessment items to provide a link to PIAAC, the 
analyses evaluated whether the ESO items worked similarly for the PISA YAFS sample as for the ESO 
field test (which was used to create the final ESO item parameters). During the IRT scaling stage, the item 
parameters were fixed to the values used in the ESO based on a method known as fixed item parameter 
linking. This permitted the item fit statistics based on the mean deviation (MD) and the root mean square 
deviation (RMSD) to be used to evaluate the performance of ESO item parameters in PISA YAFS. Poorly 
fitting items, or item misfits, indicate that these items relate differently to the skills in PISA YAFS than in 
ESO. Both item fit statistics (MD and RMSD) quantify the magnitude and direction of deviations in the 
observed data from the estimated item characteristic curve (ICC) for each single item. While the MD is 
most sensitive to the deviations of observed item difficulty parameters from the estimated ICC, the 
RMSD is sensitive to the deviations of both the observed item difficulty parameters and item slope 
parameters. Item misfits were identified using a RMSD > 0.15, and a MD > 0.15 and < -0.15 criterion, 


where a value of 0 indicates no discrepancy (in other words, a perfect fit of the model). 


Based upon data of the 1,885 respondents, three items were flagged as misfitting items. Estimating the 
new item parameters for these items would eliminate the potential bias to the linkage to the PIAAC scale, 
yet contribute to reduce measurement errors. Because these items showed misfit, it was considered 
inappropriate to use the established ESO-PIACC item parameters and therefore, unique item parameters 
had to be estimated in a separate step. Because unique parameters were only needed for a few items and 
because the common and unique item parameters were estimated to be on the same latent scale, the 
linkage and comparability was maintained across the two studies through most items sharing common 


item parameters. 


Most of the items had 5 or more responses and were used to evaluate fit of item parameters. However, 
three items in literacy had less than 5 observations due to the multistate adaptive design, and were not 
included in the evaluation. The three items are shown in table 16. All numeracy items had sufficient 


number of responses for the evaluation. 
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Table 16. Items with insufficient number of responses for the IRT parameters evaluation: PISA 


YAFS 2016 
Weighted number of 
Domain Item observations 
Literacy (3 items) D311701S 4.91 
E321001S 4.9] 
E321002S 4.9] 


Numeracy (0 items) — — 


a a RS, ca OS a a a 
SOURCE: U.S. Department of Education, National Center for Education Statistics, PISA Young Adult Follow-up Study (PISA 
YAFS), 2016. 


The three items in literacy belong to a particular testlet. Because of the multistage adaptive testing design, 
only five of the PISA YAFS respondents were assigned to this testlet (i.e., literacy testlet L1-1). The 
assignment of the first literacy testlet (L1-1, L1-2, or L1-3) and the first numeracy testlet (N1-1, N1-2, or 
N1-3) is based on a preliminary estimate of the test taker’s proficiency score, calculated using responses 
to the BQ in combination with information on how test takers with similar backgrounds performed on 
PIAAC (OECD 2015). Because the threshold values for literacy and numeracy are different (250 for 
literacy, 255 for numeracy), there is no one-to-one relationship between the assignment to literacy and 
numeracy testlets. In other words, if someone is assigned to testlet L1-1, he or she is not necessarily 
assigned to testlet N1-1. The design and test flow of the ESO is fully described in the international ESO 
technical documentation found at https://www.oecd.org/skills/ESonline-assessment/assessmentdesign/ 


technicaldocumentation/ESOnline_ Technical Doc _Ch1-8 Oct2016.pdf. 


Although these three items could not be included in the IRT scaling to evaluate the item fit (due to the 
limited sample sizes), they were included in the population modeling with the ESO item parameters. In 
most cases, the items in the PISA YAFS ESO were accurately described by the already established ESO 
item parameters previously developed using PIAAC. Only two literacy items (out of 37 included in the 
analysis from 40 in total) and one numeracy item (out of 38) needed a unique item parameter in the PISA 
YAFS data (Literacy: item C405S002 and item C407S003; Numeracy: item C602503S). Thus, results 
demonstrated that establishing comparability between PISA YAFS and ESO, as well as the PIAAC main 
study, was successful: the data from PISA YAFS can be described on a common scale with ESO and 
PIAAC. Additional information about the item content of the ESO is described in the international ESO 
technical documentation at https://www.oecd.org/skills/ESonline-assessment/assessmentdesign/ 
technicaldocumentation/ESOnline_ Technical Doc _Chl-8 Oct2016.pdf. 
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6.3 Population Modeling and Generating Plausible Values 


The population or conditioning model used for PISA YAFS was a combination of the IRT models 
described above and a latent regression model using information from the PISA 2012 student 
questionnaire (OECD 2017; von Davier and Sinharay 2013) with the goal to generate PVs. The posterior 
distribution of the ability variable from which PVs were drawn assumed to depend not only on the item 
responses on cognitive assessments but also on a large number of student questionnaire predictors (e.g., 


gender, country of birth, education, occupation, employment status, reading practices). 


In the item calibration stage, an IRT model was used to estimate item parameters for items in each 
cognitive domain (e.g., literacy and numeracy), and unique item parameters were estimated for a few 
items only (see above). Then, in the conditioning or population modeling stage, a multidimensional latent 
regression model for the domains literacy and numeracy was applied to the data, using the item 


parameters obtained in the item calibration stage and principal components based on BQ variables. 


Usually, a considerable number of background variables (predictors) are collected in large-scale 
assessments such as PISA and PIAAC, and a principal component analysis is conducted to extract the 
components that explain substantial portion of the variation for further analysis. In alignment with PIAAC 
(OECD 2013), the background variables in PISA YAFS were contrast coded; a list of variables that were 
contrast-coded and included in the population modeling is provided in appendix C. The increased number 
of variables obtained through contrast coding is substantial and needs to be reduced through a principal 
component analysis, which captures most of the common variance in the contrast-coded background 
questions but offers a reduced set of variables. The use of principal components also serves to retain 
information for examinees with missing responses to one or more background variables. In PISA YAFS, 
68.5 percent of explained variance was used to avoid numerical instability due to potential over- 
parameterization, while pertaining to the 1/20 ratio to balance the number of principal components given 
the sample size (i.e., at least unweighted 20 respondents observed for one principal component). For more 
details about the population model and the computational procedure, please see the technical report for 
the PIAAC 2012 main study at https://www.oecd.org/skills/piaac/_Technical%20Report_170CT13.pdf. 


As in the PIAAC main study, a minimum of five attempted items per domain is necessary to assure 
sufficient information about the proficiency of respondents. Thus, a two-step procedure was taken: in the 
first round, respondents who responded to at least five items within at least one domain (N = 1,868) were 
used to fit the multidimensional latent regression models, and in the second round, all respondents, 
including those who responded to fewer than five items (N = 17), received PVs fixing the regression 


parameters to the ones obtained from the first run. This procedure ensures that the cases with fewer item 
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responses did not contribute to the estimation of the proficiency distribution, but did receive the PVs in 


the domain that they responded to. 


Based on the latent regression estimates (variance-covariance matrix and regression parameters), 
multidimensional PVs (Mislevy, Johnson, and Muraki 1992; von Davier, Gonzalez, and Mislevy 2009) 
were generated and then transformed on the reporting scale that is linked to other assessments (ESO, 
PIAAC). The generated PVs were transformed on the PIAAC scale using the same transformation 
constants. In PISA YAFS, 10 independent PVs were drawn for each respondent in each cognitive domain. 
Each set of PVs was equally well designed to estimate population statistics (such as group means and 
standard deviations); however, multiple PVs were required to represent the uncertainty in the domain 
measures appropriately (von Davier, Gonzalez, and Mislevy 2009). The statistics based on scores were 
always computed at population or subpopulation levels and should never be used to draw inferences at the 
individual level. For information on how to use PVs for secondary analysis, please see the technical report 
for the PIAAC 2012 main study at 
https://www.oecd.org/skills/piaac/_Technical%20Report_170CT13.pdf. 
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7. Data Management 


The PISA YAFS data were collected online via a secure web survey that linked to the Education and 
Skills Online (ESO). Data were extracted from the website and imported into the Data Management 
Expert software. The data were given validation checks, were reconciled, and data files were prepared. 
After the dataset was assembled, a review of the confidentiality procedures for PISA 2012 relative to the 
data collected in PISA YAFS was undertaken to ascertain the procedures for evaluating disclosure risk 


and the treatments that would need to be applied to the PISA YAFS data. 


7.1 Data Cleaning and Validation 


Ensuring consistent data involves checking for valid responses and basic agreement among responses. 
Data cleaning and validation checks were not guided by previous cycle precedence because this is the first 
PISA YAFS data collection. PISA YAFS data were examined for consistency between the ESO and PISA 
2012 background data and cleaning and validation checks were employed. When in question, PISA 2012 
published data were maintained. For example, gender information from 2012 overwrote discrepant ESO 


gender values. 


Open text responses to PISA YAFS questions were reviewed and minimally edited (such as back-coding, 
correction of spelling, or interpretation of anagrams or abbreviations). From the PISA YAFS field test 
review with NCES, it was decided that the field of study open-text response (B_QO2cUSX) would not be 
coded. A treatment to truncate and shorten responses was applied and then reviewed and approved by 
NCES. 


Westat validated respondents by considering discrepant data, such as age and gender mismatch with PISA 


2012 along with some other indicators such as “Bq_q4 - born in country.” The data were reviewed for 


unrealistic performance differences to have some basic check for respondent authenticity. 


7.2 Reducing the Risk of Data Disclosure 


Since the sample design of PISA YAFS has no new strata or identifying design information and does not 
include refreshment of the sample with new respondents, there is nothing that can be used in the sampling 


design for PISA YAFS that would cause a disclosure risk for institutions or individuals. Since the sample 
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design has no new strata or identifying design information, and does not include refreshment of the 
sample with new respondents, there is nothing that can be used in the sampling design for YAFS that 
would cause a disclosure risk for institutions or individuals. No new information on the PISA sample is 
included in the final data. Only the new weights based upon the nonresponse adjustments to the original 
PISA 2012 weights will differ from the original PISA 2012 sampling and weights data. 


Westat conducted the disclosure analysis by following the DRB-approved procedures described in the 
Disclosure Analysis Plan. These procedures minimize disclosure risk for data dissemination to a level 
acceptable to NCES. That is, PISA 2012 data were examined and made confidential by implementing a 


number of procedures prior to their release: 


5 IDs (school and student) were randomly assigned with no link to the original schools 
and students; 


5 Direct identifying variables were suppressed; 


a PISA YAFS data were matched against the publicly available databases (CCD and 
PSS). Any school identified as a potential risk based on the “Rule of 3” confidentiality 
standard of NCES was masked/perturbed in order to longer be a close match with the 
CCD/PSS school. We then conducted the requisite confidentiality step of random 
swapping (DataSwap) on a set of variables identified in the DAP. This step adds 
another layer of uncertainty that a school or individual could be identified; 


7 Weights and sampling variables were analyzed to determine if institutions could be 
identified. These variables were deemed safe for PISA 2012 national data release; 


5 A set of variables from the PISA files were randomly swapped using a swapping rate 
designated by the Institute for Education Science Disclosure Review Board (DRB) 
chair; and 


5 The DRB reviewed the procedures and results of the procedures that were 


implemented for statistical disclosure control and approved for the PISA 2012 data 
release. 
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8. The PISA-YAFS 2016 Data 


Data collected in the United States for the PISA YAFS are available as a public-use file and can be 
downloaded from the NCES website at https://nces.ed.gov/surveys/pisa/datafiles.asp. The PISA YAFS 


files include data for the national sample in the United States only for those respondents that responded to 
the PISA YAFS survey, and they do not include data for PISA 2012. The PISA 2012 data are a separate 
dataset and can also be downloaded at the NCES website at https://nces.ed.gov/pubsearch/ 


getpubcats.asp?sid=098 (alternatively, they can be found under the “Data Products” section of the NCES 
PISA website). The PISA YAFS data are described in this chapter. 


8.1 PISA YAFS Dataset 


The PISA-YAFS public-use dataset includes several files to provide information to users and help them 


navigate the data. These files are listed below: 


a The data are contained in ASCII file PISA YAFS2016_Data.dat. This file contains 
questionnaire items, derived variables, and index scores based on the Learning 
Experience Questionnaire (LEQ), the Education and Skills Online (ESO) core 
questionnaire, and ESO cognitive and noncognitive items; plausible values for the 
literacy scale, the numeracy scale from the assessment; and student nonresponse 
adjusted PISA sampling weights and replicate weights. There are 2,318 cases in this 
file. Since the data are of respondents who took PISA in 2012, each record contains 
identification variables that enable the user to merge the data with the PISA 2012 
student data, using the variable STIDSTD and with the PISA 2012 school data, using 
the variable SCHOOLID. 


a An SPSS syntax file, PISA YAFS2016_SPSS.SPS, to read the ASCII file into SPSS. 

a A SAS syntax file, PISA YAFS2016_SAS.SAS, to read the ASCII file into SAS. 

5 A codebook file (PISA_ YAFS2016_Codebook.HTML) that includes variable names, 
variable location, format information, variable labels, question text, values, and 


frequencies. 


= A Read me document (PISAYAFS2016 README PUBLIC_USE.doc) that lists the 
file names associated with the public-use data set. 


8 A Quick Guide (PISA_YAFS2016 QuickGuide.doc) that lists the public-use data file 
contents, how to create working files, and a data-use agreement. 
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0 A record layout file (Layout-Y AFS2016.pdf) that provides the variable sequence, 
variable name, column position, variable type, field width, number of decimals, and 
variable label. 


C Illustrative code for SPSS and SAS to merge PISA YAFS and PISA 2012 datasets 
(Merging Code Illustrative for PISA _YAFS2016 Public Use Data.doc). 


8.2 Variable Names 


The variable names do not necessarily correspond with the question numbers in the LEQ and ESO 
instruments. For example, the LEQ variable names refer to PIAAC source variables, which are the 
original sources of the LEQ items. For convenience, variable item numbers are listed next to each item on 


the questionnaires (see appendixes A and B for the questionnaire items). 


8.3 Derived Variables 


Several derived variables were created for use in analyses, and these variables have been included in the 
PISA YAFS files. They appear after the ESO noncognitive variables measuring skill use, beginning with 
GOODFITJOB_1 and continuing through SKILLUSE_WRITING_WORK. Explanations of these 
variables are included below and are abstracted from Education and Skills Online technical 
documentation (OECD 2015). Section 8.3.1 describes measures related to skill use, section 8.3.2 
describes measures derived from the Career Interest and Intentionality module, and section 8.3.3 


describes measures derived from the subjective well-being items. 


8.3.1 Skill Use 


The Skill Use module for Education & Skills Online includes 57 items across 8 scales. The scales focus 
on the frequency with which respondents use the skills associated with reading, writing, numeracy, and 


information and communications technology (ICT) at home and at work. 


The reading scale measures how often respondents use the skills required to read documents such as 
directions, instructions, letters, memos, e-mails, articles, books, manuals, bills, invoices, diagrams, and 
maps. The writing scale measures how often respondents use the skills required to write documents such 
as letters, memos, e-mails, articles, reports, and fill-in forms. The numeracy scale measures how often 


respondents use the skills required to calculate prices, costs, or budgets; use fractions, decimals, or 
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percentages; use calculators; prepare graphs or tables; use algebra or formulas; and use advanced math or 
statistics. The ICT scale measures how often respondents use the skills required to use e-mail, the 
Internet, spreadsheets, word processors, and programming languages; conduct transactions online; and 


participate in online discussions (conferences, chats). 


For each scale, respondents are asked four to eight questions about how often they use these skills in their 
home or work lives. The response options are (1) never, (2) less than once a month, (3) less than once a 
week but at least once a month, (4) at least once a week but not every day, and (5) every day. For ICT 
skill use, respondents were first asked whether they had ever used a computer; questions assessing the 
domain are not presented to those without any previous contact with computers. In contrast, reading, 
writing, and numeracy skills used at home are assessed for all respondents. The corresponding scales for 
skills used at work are assessed only for those respondents who are part of the labor force or have been in 
the labor force at some time, as determined by their answer to the background question on their current 
employment status and a question at the beginning of the Skill Use module asking if they have ever been 


employed. 


Respondents receive scores of not applicable, low, moderate, or high for each of the skill use scales, 


defined as follows: 


a Not applicable: The respondent reported that he or she never engaged in any of the 
activities involving this skill. 


7 Low: The respondent reported that he or she rarely engaged in most of the activities 
involving this skill. 


c Moderate: The respondent reported that his or her engagement in activities varied in 
terms of how many activities were done and how often they were done. 


7 High: The respondent reported that he or she engaged in most activities on most days 
or every day. 


Respondent receive a score of not applicable when they indicate that they have never engaged in any of 
the activities mentioned in the module for that scale. To determine whether a respondent should receive a 
score of low, moderate, or high for a particular skill, the respondent’s responses are compared to 
responses from participants in the 24 countries in the Program for the International Assessment of Adult 
Competencies (PIAAC). Test takers receive a score of low if their responses indicate they are in the 
bottom quintile (one-fifth of the distribution) of individuals internationally who use that skill. Test takers 


receive a score of moderate if their responses indicate they are in the middle three quintiles of individuals 
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internationally who use that skill. Test takers receive a score of high if their responses indicate they are in 


the top quintile of individuals internationally who use that skill. 


8.3.2 Career Interest and Intentionality Items and Related Indices 


Career Interest Areas: PERSON_A_TOTAL—PERSON_S_ TOTAL 


The Career Interest and Intentionality module consists of 60 items from the O*NET Interest Profiler Short 
Form (Rounds et al. 2010). This set of items is composed of 10 items from each of the six RIASEC scales 
(realistic, investigative, artistic, social, enterprising, and conventional) (Holland 1997). All items had a 
five-point Likert response scale from 1) strongly dislike to 5) strongly like. Scores for each RIASEC 
dimension are calculated by averaging the 10 item values within each dimension. The respondents 
received a score of 0 to 40 in each interest area. Table 17 describes the interest areas. Higher scores 


indicate the respondent’s interests are more aligned with that interest area. 


Table 17. Interest area description 


Interest area _ Description Examples of work 

Realistic People with realistic interests like work that = Working with plants and 
includes practical, hands-on problems and animals 
answers. Often people with realistic interests do = Real-world materials like 
not like careers that involve paperwork or wood, tools, and machinery 
working closely with others. m= Outside work 

Investigative People with investigative interests like work that m= Searching for facts 
has to do with ideas and thinking rather than = Figuring out problems 
physical activity or leading people. 

Artistic People with artistic interests like work that deals = Creativity in their work 
with the artistic side of things, such as acting, = Work that can be done without 
music, art, and design. following a set of rules 

Social People with social interests like working with m= Teaching 
others to help them learn and grow. They like m Giving advice 
working with people more than working with = Helping and being of service 
objects, machines, or information. to people 

Enterprising People with enterprising interests like work that m Persuading and leading people 
has to do with starting up and carrying out m= Making decisions 
business projects. They like taking action rather m Taking risks for profits 


than thinking about things. 

Conventional People with conventional interests like work that == Working with clear rules 
follows set procedures and routines. They prefer m Following a strong leader 
working with information and paying attention to 
details rather than working with ideas. 


SOURCE: U.S. Department of Education, National Center for Education Statistics, PISA Young Adult Follow-up Study (PISA 
YAFS), 2016. 
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Job Interest: JOB_FIT_CURRENT_CATEGORIZED and 
JOB_FIT_INTENDED_ CATEGORIZED 


The career interest assessment uses the interest profile for 436 occupations from the O*NET database to 
determine how well the respondent’s interests match the interest profile of the respondent’s current and 
desired occupations. Using the respondent’s RIASEC profile, a job fit score from minus-100 to 100 is 
calculated for the current and desired occupations. The occupation is considered a low fit if the job fit 
score is less than 10 points, a moderate fit if the score is between 10 and 50 points, and a high fit if the 
score is 50 points or above. If the respondent indicates in the background questionnaire that he or she is 
“unemployed, not looking for work” then no score will be provided for the fit of current and desired jobs. 
A job fit score also is calculated for each of the 436 occupations, and a list of the highest scoring 20 
occupations and the lowest scoring 10 occupations are provided for the respondent. The highest scoring 
occupations are considered the best fit for the respondent’s interests and the lowest scoring are considered 


the worst fit. 


Job Fit: GOODFITJOB and POORFITJOB indices 


Occupational interest profiles in O*NET were developed using subject matter expert ratings. Two groups 
of three trained raters considered those occupations included in the O*NET database. The appropriateness 
of each RIASEC category for each occupation based on O*NET data for the occupation was evaluated. 
The mean rating for the three reviewers was calculated for each of the six interest dimensions across 
occupations. Inter-rater agreement and validity evidence were also assessed. A high degree of rater 


reliability was found, as was alignment to Holland’s theoretical RIASEC model. 


Job Seeking and Training Intentionality: SEEKJOB_GROUP, SEEKTRAINING_GROUP, 
SELFEFFICACY_GROUP, TAKEACTIVE_ STEPS GROUP 


The career intentionality portion of the assessment consists of 26 items.* This set of items is composed of 
six items that measure job-seeking intentionality, six that measure training intentionality, four that 
measure job-seeking and training self-efficacy, and 10 that measure taking active steps. Job-seeking 


intentionality, training intentionality, and job-seeking and training self-efficacy scales had a six-point 


4 These items are not included in the PISA YAFS 2016 dataset. The items are proprietary and cannot be released publicly. 
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response scale from 1) strongly disagree to 6) strongly agree. Scores are calculated for each scale by 
averaging item responses. The “taking active steps” scale had a binary response of yes or no. The total 
number of yes responses is used as the score. If the test taker indicates in the background questionnaire 
that he or she is “unemployed, not looking for work” then no score was provided for the career 


intentionality assessment. 


The job-seeking intentionality, training intentionality, and job-seeking and training self-efficacy scales 
use stanine (nine-point standard scale) scores to determine low, moderate, and high scoring groups. 
Scores were placed on the scale, using lower and upper bounds to establish a range for each group (note: 
stanines 8 and 9 were combined due to small N-count). Cutoffs for low, moderate, and high are included 
in Table 18. Though gaps existed in the values of the upper and lower bounds, none of the calculated 
intentionality scores will equal those values. For the taking active steps scale, low (~50 percent of norm 
group; score = 0), moderate (~25 percent; score between | and 3), and high groups (~25 percent; score 


between 4 and 8) were established using test takers’ raw scores from the Field Test. 


Table 18. Career Intentionality score cutoffs 


Score 
Scale Low Moderate High 
Job-seeking intentionality 1.00 — 1.67 1.83 — 4.84 5.00 — 6.00 
Training intentionality 1.00 — 2.67 2.83 — 5.00 5.16 — 6.00 
Job-seeking and training self-efficacy 1.00 — 3.25 3.50 — 5.25 5.50 — 6.00 
Taking active steps 0 1.00 — 3.00 4.00 — 8.00 


5] 
SOURCE: U.S. Department of Education, National Center for Education Statistics, PISA Young Adult Follow-up Study (PISA 
YAFS), 2016. 


8.3.3 Subjective Well-Being and Health 


The subjective well-being portion of the module focuses on a respondent’s attitudes and feelings toward 
his or her life, using cognitive and emotional measures of life satisfaction. The cognitive measure is an 
adapted version of the Satisfaction with Life Scale (SWLS) (Diener et al. 1985). The adapted SWLS 
includes four items on a six-point Likert type response scale including 1) strongly disagree, 2) disagree, 3) 
slightly disagree, 4) slightly agree, 5) agree, and 6) strongly agree. Scores are calculated by averaging the 
four item responses, resulting in a total score from | to 6, and then comparing them to the scores collected 
during the Field Test. A quartile (one-fourth of the distribution) approach, appropriate for cross-cultural 
comparisons, was used to define scoring cutoffs for reporting. Scores in the first quartile are reported as 


low, in the second and third quartiles as moderate, and in the fourth quartile as high. 
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Positive and Negative Affect: POSITIVE_AFFECT_LABEL and NEGATIVE_AFFECT_LABEL 


The second element of subjective well-being is an emotional evaluation describing the respondent’s 
emotional experience of his or her life. While life satisfaction is assessed on a single dimension, the 
emotional evaluation is composed of two distinct dimensions: positive affect (PA) and negative affect 
(NA). The Education & Skills Online measure for emotional evaluation is an adapted version of the 
Positive and Negative Affect Schedule (PANAS) (Watson, Clark, and Tellegen 1988) and I-PANAS-SF 
(Thompson 2007), an internationally validated short form of the instrument. The Education & Skills 
Online scale is composed of nine items, including four PA items and five NA items. Respondents are 
asked to rate their experience of each emotion during the previous week, measured using a five-point 
scale including 1) very slightly or not at all, 2) a little, 3) moderately, 4) quite a bit, and 5) extremely. 
Scores for PA are calculated by averaging the four positive item responses, while NA is calculated by 
averaging the five negative item responses, resulting in one total score for each dimension. Dimension 
scores are then compared to the 1,890 scores collected during the Field Test. A quartile approach, 
appropriate for cross-cultural comparisons, is used to define scoring cutoffs, reporting PA and NA scores 


in the first quartile as low, in the second and third quartiles as moderate, and in the fourth quartile as high. 


Health is a complex multidimensional construct whose definition has evolved from a purely biological 
measure to include psychosocial factors considered critical to the assessment of overall well-being. 
Gathering health data is an integral component of the ongoing effort to monitor economic and social 
progress across countries and promote policies aimed at improving overall life quality (OECD 2011). The 
Education & Skills Online measures of subjective and behavioral health include 14 survey items on the 
feelings and behaviors most relevant to health as described in the OECD agenda. These include items on 
subjective health, body mass index (BMJ), nutrition, exercise, sleep, and smoking status. Reporting of 
subjective and behavioral health is presented in an informational format as the respondent’s self-reported 
health perceptions and behaviors are compared to the accepted international health recommendations 
(World Health Organization; [WHO 2015]) for each category. 


Subjective Health: SUBJ_-HEALTH_LABEL 


Subjective health is a single item measure on a respondent’s self-perception of his or her health, measured 
with a six-point response scale, including 1) very poor, 2) poor, 3) fair, 4) good, 5) very good, and 6) 
excellent. Based on the response, a health outlook is reported as poor for scores | or 2, fair for 3 or 4, and 


positive for 5 or 6. 
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Body Mass Index (BMI): BMI_CLASS 


BML, an internationally accepted health measure, is calculated using the self-reported responses for height 
and weight. The corresponding report element offers a definition of BMI as well as a classification of 
underweight, normal weight, or overweight based on the international classification of the WHO (WHO 
2015). 


Diet Score: DIET_LABEL 


As an indicator of nutrition and based on the international nutritional recommendations (WHO 2015), 
four questions elicit the number of servings per day and days per week that the respondent consumes 
fruits and vegetables (DIET_LABEL). Self-reported servings and frequencies of fruit and vegetable 
consumption are used to provide an evaluation of the respondent’s diet. The corresponding report element 
on diet and nutrition details the current recommendation and indicates whether the respondent consumes 
no fruits and vegetables, insufficient amounts of fruits and vegetables, or greater than or equally sufficient 


amounts of fruits and vegetables per the current recommendations. 


Smoking: SMOKE 


One item elicits the smoking status of the respondent by asking if he or she currently smokes any tobacco 
products including cigarettes, cigars or pipes, offering response options of no, yes, sometimes, or “yes, 
daily.” Because smoking is a recognized health risk and not recommended at any level, the corresponding 
report element describes the health risk of smoking and indicates whether the respondent has reported 


smoking behavior. 


Physical Activity Score: PHYSICAL_ACTIVITY_LABEL 


Based on the international recommendations (WHO 2015) for physical activity, which suggest 75 minutes 
of intense activity or 150 minutes of moderate activity per week, four items eliciting information on the 
frequency and intensity of physical activity are included. These four items request the amount of time and 
number of days a respondent engages in both moderate and vigorous exercise. Total times are calculated 
for both moderate and intense activity and equated to a common scale (weekly minutes of intense exercise 


times 2), which is compared to the 150 minutes recommendation. The corresponding report element on 
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exercise details the current recommendation and indicates whether the respondent does not exercise, 


exercises at a level insufficient to the stated standard, or meets or exceeds the recommendation. 


Sleep Categorization: SLEEP_LABEL 


Two items elicit duration and quality of sleep, as these are core features of commonly accepted sleep 
recommendations. The item on sleep quality offers responses of very bad or fairly bad for indicators of 
insufficient sleep quality, and fairly good or very good as indicators of adequate sleep quality. The sleep 
duration item requests the average amount of sleep in hours for the past month. The health behavior report 
details the current recommendation of seven to nine hours of quality sleep and related health benefits 


while reporting whether the test taker meets the recommendation. 


8.3.4 Adapted Variable 


Question B-Q02bUS asked, “What type of degree or certificate are you currently studying for?” with the 
following categories: Grades 1-6; Grades 7-9; High school diploma; Pre-associate education; Attended 
trade school, college, or university; no certificate or degree received; A certificate from a college or trade 
school for completion of a program prior to the associate/bachelor’s degree; Associate degree; Bachelor’s 
degree (e.g., BA, AB, BS); Master’s degree (e.g., MA, MS, MEng, MEd, MSW, MBA); Professional 
degree (e.g., MD, DDS, DVM, LLB, JD); or Doctorate degree (e.g., PhD, EdD). See appendix A for the 


full question format. 


The categories Professional degree and Doctorate degree were cells that were too small to be useful for 
analysis and could potentially be identifying if a student’s school were known (though this is improbable 
given the data available). These categories were collapsed to a single category, Professional/Doctorate 


degree. 
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8.4 Missing Value Codes 


Table 19 provides missing value codes in the SAS and SPSS datasets. 


Table 19. Missing value codes in SAS and SPSS datasets 


Missing value code Text data file missing SAS missing value SPSS missing value 
description value code code code 
N/A 0 0 0 
ESO nonrespondent 9995 9995 9995 
Valid skip 9996 9996 9996 
Missing 4 blanks : . 

Not answered 1 to 17 blanks 

depending on width of 

variable 


. . 
SOURCE: U.S. Department of Education, National Center for Education Statistics, PISA Young Adult Follow-up Study (PISA 
YAFS), 2016. 


8.5 Special Considerations 


Three aspects of the PISA YAFS design need careful attention in any analysis. The first stems from the 
sample design. The use of sampling weights is necessary for the computation of statistically sound, 
nationally representative estimates because simple random sampling was not employed. Although 
respondents had known probabilities of selection, these probabilities were unequal. Adjusted survey 
weights adjust for the probabilities of selection for nonresponse. Thus, to generalize to the population, 


sampled analyses will need to apply the sampling weights provided in the file. 


The second aspect to be considered also stems from the sampling design and involves the calculation of 
standard errors. Since the sample design is complex (a two-stage, stratified cluster design), most software 
packages, operating on the assumption of a simple random sample, will produce biased estimates of 
standard errors. To use the replicate weights contained in the data file, one must use special procedures to 
produce unbiased estimates of the standard errors. These procedures involve the use of Fay’s method of 
balanced repeated replicates (BRR) with 80 replicates and the Fay coefficient set to 0.5 to estimate the 
standard errors. Further descriptions can be found in the PISA Data Analysis Manual: SPSS, Second 
Edition (OECD 2009). 


The third aspect arises from the design of the PISA YAFS performance variables and the use of PVs in 


the analysis. While the data has individual scores for respondents on ESO cognitive items, in PISA 


YAFS, as in many national and international assessments, respondents are not administered every 
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assessment item. Each item has missing responses, though these are missing by design. Thus, it is not 
possible to estimate scores for individuals. Instead, the results of individual respondents are aggregated to 
produce a set of scores for groups of respondents (e.g., U.S. female students). The distribution of scores 
indicates a set of plausible values, which represent a range of abilities for a certain group of students. For 
analysis purposes, the PISA YAFS dataset includes a set of 10 plausible values for each of the PISA 
YAFS scales. Thus, if any analysis were to be undertaken with any of the PISA YAFS scales, it should be 
undertaken 10 times, once for each plausible value. The results would then be averaged, and any 
significance tests would have to be adjusted for variation between the first 10 sets of results. A special 
provision also needs to be made in estimating the standard errors and is best done using the appropriate 
software developed for this purpose (EdSurvey; WesVar), which is described next. Further discussion of 
the use of PVs can be found in the PISA Data Analysis Manual: SPSS, Second Edition (OECD 2009) 


available at https://www.oecd.org/pisa/pisaproducts/pisadataanalysismanualspssandsassecondedition.htm. 


EdSurvey is an R statistical package designed for the analysis of national and international education data 
from NCES, including the PISA YAFS data. 


How to install and load EdSurvey: 


1. Inside R, run the following command to install EdSurvey as well as its package 
dependencies: 


linstall_packages(“EdSurvey”) 


2. Once the package is successfully installed, EdSurvey can be loaded with the following 
command: 


library(EdSurvey) 


For additional information on EdSurvey, visit: https:/Awww.air.org/project/nces-data-r-project-edsurvey. 


An alternative to EdSurvey is WesVar. WesVar is a Statistical software package used to compute 
estimates and replicate variance estimates from complex survey data. WesVar utilizes a user interface 
allowing users to create weights, specify tables, and define regression models. Users can create survey 
weights using three methods of jackknife replication and two versions of balanced repeated replication. 
Users select the variables that identify strata and primary sampling units and choose a replication method. 
WesVar creates a set of weights for each replicate subsample. Users can then adjust the weights for 


nonresponse, poststratify the weights, or rake weights to control totals. 
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With WesVar, you can perform the following functions: 


7 Import data from many types of files, including SAS, SPSS, Excel, text files, or other 
data files that comply with the Object Database Connectivity (ODBC) standard; 


a Create replicate weights if they are not already on the file; 
rT Recode, relabel, and reformat variables; 
7 Calculate estimates and standard errors for means, totals, and other statistics in cells 


and marginals of multiway tables; 


a Calculate standard errors for complex transformations of variables and for linear and 
logistic regression parameters; 


7 Organize your analyses in easy-to-use, tree-structured workbooks; 
7 Control and customize the output; and 


a Export data files to SPSS, SAS Transport, or ASCII formats. 


WesVar is available free of charge from Westat at https://www.westat.com/capability/ 


information-systems-software/wesvar. 


PISA YAFS data can also be used with the International Association for the Evaluation of Educational 


Achievement (IEA) International Database Analyzer (IDB Analyzer) (https://www.iea.nl/data-tools/ 


tools#spy-para-308). The IDB Analyzer is a free software tool that can be used to combine and analyze 


data from many large-scale assessments. In order for the PISA YAFS file to be read into the IDB 
Analyzer, the user has to add and rename a few variables to match the PISA YAFS pre-2015 file 
structure, which are outlined below: 


7 Variable CNT, for a country, has to be created and assigned a string value of “USA.” 


rT Variables W_YFSTR1 to W_YFSTR80, i.e., replicate weights, have to be renamed 
W_FSTR1 to W_FSTR80. 


a Variable W_YFSTUWT, i.e., final sample weight, has to be renamed to W_FSTUWT. 


When the estimates are run on the ESO plausible values only, it is recommended to change the SPSS/SAS 
code to pick up all 10 plausible values. 


Other commercial packages that include support for the weighting and replication methods used in PISA 
YAFS, among others, are SAS 9.4, SUDAAN 11 (2013), and Stata 13 (2013). 


PISA YAFS Technical Report and User Guide 62 


References 


Birnbaum, A. (1968). Some Latent Trait Models and Their Use in Inferring an Examinee’s Ability. In 
F.M. Lord and M.R. Novick (Eds.), Statistical Theories of Mental Test Scores. Reading, MA: 
Addison-Wesley. 


Diener, E., Emmons, R.A., Larsen, R.J., and Griffin, S. (1985). The Satisfaction With Life Scale. Journal 
of Personality Assessment, 49: 71-75. 


Holland, J.L. (1997). Making Vocational Choices: A Theory of Vocational Personalities and Work 
Environments (3rd ed.). Odessa, FL: Psychological Assessment Resources. 


Mislevy, R.J., Johnson, E., and Muraki, E. (1992). Chapter 3: Scaling Procedures in NAEP. Journal of 
Education and Behavioral Statistics, 17(2): 131-154. 


Organization for Economic Cooperation and Development (OECD). (2009). PISA Data Analysis Manual: 
SPSS (2™ ed.). Paris: Author. 


Organization for Economic Cooperation and Development (OECD). (2011). How’s Life?: Measuring 
Wellbeing. Paris: Author. 


Organization for Economic Cooperation and Development (OECD). (2012). Literacy, Numeracy, and 
Problem Solving in Technology-Rich Environments: Framework for the OECD Survey of Adult 
Skills. Paris: Author. Retrieved February 5, 2018, from 


https://www.oecd.org/skills/piaac/PIAAC%20Framework%202012-- 
%20Revised%20280ct2013_ebook.pdf. 


Organization for Economic Cooperation and Development (OECD). (2013). Technical Report of the 
Survey of Adult Skills (PIAAC), Ch. 17 (pp. 406-438). Paris: Author. Retrieved January 2019, from 
https://www.oecd.org/skills/piaac/_Technical%20Report_170CT13.pdf. 


Organization for Economic Cooperation and Development (OECD). (2014). PISA 2012 Results: What 
Students Know and Can Do — Student Performance in Mathematics, Reading and Science (Volume 
I, Revised edition, February 2014). Paris: Author. 


Organization for Economic Cooperation and Development (OECD). (2015). Education & Skills Online 
Technical Documentation. Paris: Author. Retrieved January 2019, from 
https://www.oecd.org/skills/ESonline-assessment/assessmentdesign/technicaldocumentation/. 


Organization for Economic Cooperation and Development (OECD). (2017). PZSA 2015 Technical Report. 
Paris: Author. Retrieved January 2019, from: 


https://www.oecd.org/pisa/data/2015-technical-report/. 

Rao, J.N.K., and Thomas, D.R. (2003). Analysis of Categorical Response Data From Complex Surveys: 
an Appraisal and Update. In R.L. Chambers and C.J. Skinner (Eds.), Analysis of Survey Data (pp. 
85-108). West Sussex, England: John Wiley and Sons LTD. 


Rounds, J., Su, R., Lewis, P., and Rivkin, D. (2010). O*NET® Interest Profiler Short Form Psychometric 
Characteristics: Summary. Raleigh, NC: National Center for O*NET Development. 


PISA YAFS Technical Report and User Guide R-1 


Thompson, E.R. (2007). Development and Validation of an Internationally Reliable Short-Form of the 
Positive and Negative Affect Schedule (PANAS). Journal of Cross-Cultural Psychology, 38: 227- 
242. doi:10.1177/0022022106297301. 


Tibshirani, R.J. (1996). Regression Shrinkage and Selection via the LASSO. Journal of the Royal 
Statistical Society, Series B, 58: 267-288. 


von Davier, M., Gonzalez, E., and Mislevy, R. (2009). What Are Plausible Values and Why Are They 
Useful? In JERI Monograph Series: Issues and Methodologies in Large Scale Assessments, Vol. 2. 
Retrieved January 2019, from: https://www.ierinstitute.org/fileadmin/Documents/ 
IERI_ Monograph/IERI_ Monograph Volume 02 Chapter 01.pdf. 


von Davier, M., and Sinharay, S. (2013). Analytics in International Large-Scale Assessments: Item 
Response Theory and Population Models. In L. Rutkowski, M. vonDavier, and D. Rutkowski 
(Eds.), Handbook of International Large Scale Assessment (pp. 155-174). Boca Raton, FL: CRC 
Press. 


Watson, D., Clark, L.A., and Tellegen, A. (1988). Development and Validation of Brief Measures of 
Positive and Negative Affect: The PANAS Scales. Journal of Personality and Social Psychology, 
54(6): 1063-1070. 


Westat. (2007). WesVar 5.0 User’s Guide. Rockville, MD: Westat. 


World Health Organization (WHO). (2015). Global database on body mass index. Retrieved August 5, 
2015, from https://www.who.int/nutrition/databases/bmi/en/ 


PISA YAFS Technical Report and User Guide R-2 


Appendix A. Learning Experience Questionnaire (LEQ) 


Table A-1. 


Learning Experience Questions (LEQ) — Additional Background Questions 


Question 


Description 


B 


We 


B_Q02cUSX 


B 


DOomMIDNARWNHE 


 Q02a 


_Q02bUS 


Q02c 


| Q04a 


Question 


Are you currently studying for any kind of formal degree or 
certificate? 


Yes 

No 

What type of degree or certificate are you currently studying 
for? 


Grades 1-6 

Grades 7-9 

High school diploma 

Pre-associate education. Attended trade school, college, or 
university; no certificate or degree received 

A certificate from a college or trade school for completion of a 
program prior to the associate/bachelor’s degree 

Associate degree 

Bachelor’s degree (e.g., BA, AB, BS) 

Master’s degree (e.g., MA, MS, MEng, MEd, MSW, MBA) 
Professional degree (e.g., MD, DDS, DVM, LLB, JD) 
Doctorate degree (e.g., PhD, EdD) 

What was the area of study, emphasis, or major for this degree 
or certificate? If there was more than one, please choose the one 
you consider most important. 

Now, looking at this card, in which category would you place 
“B_Q02cUSX? Again, if there was more than one, please 
choose the one you consider most important. 


General programs 

Teacher training and education studies 

Humanities, languages and arts 

Social sciences, business and law 

Science, mathematics and computing 

Engineering, manufacturing and construction 

Agriculture and veterinary 

Health and welfare 

Services 

During the last 12 months, that is since “MonthYear, have you 
studied for any formal degree or certificate, either full-time or 
part-time? 

Yes 

No 


Theme 


Current studies; 
level of degree; area 
of study 


Current studies; 
level of degree; area 
of study 


Current studies; 
level of degree; area 
of study 

Current studies; 
level of degree; area 
of study 


Studies in the last 12 
months 


See note at end of table. 
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Table A-1. Learning Experience Questions — Additional Background Questions—Continued 
Question 
Description Question Theme 
B R12 [TU-EN-Question-B_R12-1]We would now like to turn to 
other organized learning activities you may have participated 
in during the last 12 months, including both work and non- 
work-related activities. We will distinguish between courses 
mentioned on this show card. When answering the next 
questions, please exclude any activity you engaged in as part 
of the study you already reported on.[TU-EN-Question- 
B_R12-2]We would now like to turn to other organized 
learning activities you may have participated in during the last 
12 months, including both work and non-work-related 
activities. We will distinguish between courses mentioned on 
this show card. 
B Ql2a During the last 12 months, have you participated in courses Nonformal learning 
conducted through open or distance education? activities (distance 
education, on-the- 
job courses, 
seminars, other 
private lessons) 
1 Yes 
2 No 
B_Ql2c During the last 12 months, have you attended any organized Nonformal learning 
sessions for on-the-job training or training by supervisors or activities (distance 
co-workers? education, on-the- 
job courses, 
seminars, other 
private lessons) 
1 Yes 
2 No 
B Ql2e During the last 12 months, have you participated in seminars Nonformal learning 
or workshops? activities (distance 
education, on-the- 
job courses, 
seminars, other 
private lessons) 
1 Yes 
2 No 
B Ql2g During the last 12 months, have you participated in courses or Nonformal learning 
private lessons, not already reported? activities (distance 
education, on-the- 
job courses, 
seminars, other 
private lessons) 
1 Yes 
2 No 


See note at end of table. 
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Table A-1. Learning Experience Questions — Additional Background Questions—Continued 


Question 

Description Question Theme 

[B_Q26a] [TU-EN-Question-B_Q26a-1] 

In the last 12 months, were there more learning activities you 

wanted to participate in but did not? Include both learning 

activities that lead to formal degrees and other organized 

learning activities. 

[TU-EN-Question-B_Q26a-2] 

In the last 12 months, were there any learning activities you More/any learning 

wanted to participate in but did not? Include both learning activities, wanted, 

activities that lead to formal degrees and other organized but could not 

learning activities. participate in; 
reasons for non- 
participation. 

1 Yes 

2 No 

[B_Q26b] Which of the following reasons prevented you from More/any learning 

participating in education and training? Please indicate the activities, wanted, 

most important reason. but could not 
participate in; 
reasons for non- 
participation. 

1 I did not have the prerequisites. 

2 Education or training was too expensive/I could not afford it. 

3 Lack of employer support. 

4 I was too busy at work. 

5 The course or program was offered at an inconvenient time. 

6 I did not have time because of child care or family 

responsibilities. 

7 Something unexpected came up that prevented me from 

taking education or training. 

8 Other 

[B_R27aUSX] During the past 12 months, did you take any classes or have a 

tutor. 

[B_Q27aUSX] to improve your basic reading, writing, and math skills? Basic education 
courses in the last 12 
months (reading/ 
writing/math, GED, 
some other adult 
education) 

1 Yes 

2 No 


See note at end of table. 
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Table A-1. Learning Experience Questions — Additional Background Questions—Continued 
Question 
Description Question Theme 


[B_Q27bUSX] 


1 
2 
[B_Q27cUSX] 


1 
2 
[B_Q29aUSX] 


1 
2 


to prepare to take the General Educational Development test, 
or GED? 


Yes 

No 

in some other high school equivalency program or adult high 
school program? 


Yes 

No 

During the past 12 months, were you in a formal 
apprenticeship program leading to journeyman status in a 
skilled trade or craft? 

Yes 

No 


Basic education 
courses in the last 12 
months (reading/ 
writing/math, GED, 
some other adult 
education) 


Basic education 
courses in the last 12 
months (reading/ 
writing/math, GED, 
some other adult 
education) 


Formal 
apprenticeship in the 
last 12 months 


ea ee a on ee ee ae 
SOURCE: U.S. Department of Education, National Center for Education Statistics, PISA Young Adult Follow-up Study (PISA 


YAFS), 2016. 
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Appendix B. ESO Noncognitive Items 


Table B-1. ESO Noncognitive Battery — Core Background Questionnaire 
Item 
Directions Routing Concept/ 
Item Item Text Text Rules Response Options Construct 
bq_ql How old are Use the Open-ended item Age 
you? (years number 
old) keys to 
type your 
answer. 
bq_q2 Are you female Click on Female//Male Gender 
or male? your 
answer. 
bq_q3 What is the Click on No education//Primary Education 
highest level of your education//Secondary education 
education you — answer. without a diploma//Secondary 
have Education (i.e., High School 
completed? diploma, GED or equivalent)// 
Some Post-Secondary 
Education (including 
community, technical or 
vocational)//4-year College or 
University degree//Beyond a 
College or University degree 
bq_q4 Were you born Yes//No Country of 
in [country]? Origin 
bq_q5 In what year Use the If No (2) Drop-down options: Before Country of 
did you first pull-down to bq q4 1970//1971-2013 (individual Origin 
come to livein menu to year options)//After 2013 
[country]? select your 
answer. 
bq_q6 What is the Click on [national language 1]//[national Language 
language that your language 2, if applicable]// First 
you first answer. [other common language in Learned 
learned at your country]//Other 
home in 
childhood and 
still 
understand? 
bq_q7_ Enter your If Other Open-ended item. Note, even Language 
native (4) to though this item is onthe same First 
language: bq_q6 screen as bq_q6, the response in Learned 
the text box should be codedas “Other” 
bq_q7. 
See note at end of table. 
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Table B-1. 


SO Noncognitive Battery — Core Background Questionnaire—Continued 


Item 
bq_q8 


bq_q9 


Item Text 


Which of the 
following best 
describes your 
current 
employment 
status? 


What is your 
current 
occupation (or 
intended 
occupation if 
currently 
unemployed)? 
Please select a 
broad 
occupation 
category on 
this screen and 
more specific 
occupation 
groups and job 
titles on the 
following 
screens. 


Item 

Directions Routing 

Text Rules 

Click on 

your 

answer. 
If answered 
full-time (1), 
part-time 
(2), or 
Unemployed 
and looking 
for work (3) 
to bq_q5 


Response Options 

Full-time employed (including 
self-employed)//Part-time 
employed (including self- 
employed)//Unemployed (not 
employed and looking for 
work)//Unemployed and not 
looking for work (e.g., 
student, retired, homemaker or 
permanently disabled)// 
Apprentice, intern//Other 

SEE Worksheet: ISCO-08 
structure ENg09 for US- 
English version text. PDF 
document includes mock-up of 
screens to illustrate screen 
contents/sequencing. 


Concept/ 
Construct 
Employment 
Status 


Current 
Occupation 


er a SE a eee Bs SE SL Se SE A ADI RE I RR 
SOURCE: U.S. Department of Education, National Center for Education Statistics, PISA Young Adult Follow-up Study (PISA 
YAFS), 2016. 
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Table B-2._ ESO Noncognitive Battery — Career Interest and Intentionality (CID) 


Question Description 


STEM: CII 02-61 This inventory contains a list of activities to help you explore 
your vocational interests. Please indicate how much you would 


like to do each activity by clicking on the response that most 


(102) CIL 02 
(112) CIL_12 
(122) CiL22 
(132) CIL 32 
(142) CIL_42 
(152) CIL 52 
(103) CIL 03 
(113) CIL 13 
(123) CIL 23 
(133) CIL 33 
(143) CIL 43 
(153) CIL 53 
(104) CIL 04 
(114) CIL_14 
(124) CIL 24 
(134) CIL 34 
(144) CIL 44 
(154) CIL 54 
(105) CIL_05 
(115) CIL_15 
(125) CIE 25 
(135) CIL 35 
(145) CIL 45 
(155) CIL_55 
(106) CIL_06 
(116) CIL_16 
(126) CIL 26 
(136) CI 36 
(146) CIL_46 
(156) CIL_56 
(107) CIL_07 
(117) CIL_17 
(127) CIL 27 
(137) CIL 37 
(147) CIL_47 
(157) CIL_57 
(108) CIL_ 08 
(118) CIL 18 


See note at end of table. 


closely represents how you feel about it. 
1-Strongly dislike 
2-Dislike 
3-Neutral 
4-Like 
5-Strongly Like 
Build kitchen cabinets. 
Develop a new medicine. 
Write books or plays. 
Teach an individual an exercise routine. 
Buy and sell stocks and bonds. 
Develop a spreadsheet using computer software. 
Lay brick or tile. 
Study ways to reduce water pollution. 
Play a musical instrument. 
Help people with personal or emotional problems. 
Manage a retail store. 
Proofread records or forms. 
Repair household appliances. 
Conduct chemical experiments. 
Compose or arrange music. 
Give career guidance to people. 
Operate a beauty salon or barber shop. 
Load computer software into a large computer network. 
Raise fish in a fish hatchery. 
Study the movement of planets. 
Draw pictures. 
Perform rehabilitation therapy. 
Manage a department within a large company. 
Operate a calculator. 
Assemble electronic parts. 
Examine blood samples using a microscope. 
Create special effects for movies. 
Do volunteer work at a nonprofit organization. 
Start your own business. 
Keep shipping and receiving records. 
Drive a truck to deliver packages to offices and homes. 
Investigate the cause of a fire. 
Paint sets for plays. 
Teach children how to play sports. 
Negotiate business contracts. 
Calculate the wages of employees. 
Test the quality of parts before shipment. 
Develop a way to better predict the weather. 
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Table B-2._ ESO Noncognitive Battery — Career Interest and Intentionality (CIJ)—Continued 


Question Description 
STEM: CII_02-61 — continued 


(128) CI 28 
(138) CII 38 
(148) CIL 48 
(158) CIL_58 
(109) CIL_09 
(119) CIL_19 
(129) CIL 29 
(139) CIL 39 
(149) CIL 49 
(159) CIL 59 
(110) CIL_10 
(120) CIL 20 
(130) CIL 30 
(140) CIL 40 
(150) CIL_50 
(160) CIL 60 
(111) CIL_11 
(121) CIL_21 
(131) CIE 31 
(141) CIE 41 
(151) CIL 51 
(161) CIL 61 
(188) CI 88 


(189) CIL 89 


(190) CIL 90 


See note at end of table. 


Write scripts for movies or television shows. 
Teach sign language to people with hearing disabilities. 
Represent a client in a lawsuit. 
Inventory supplies using a hand-held computer. 
Repair and install locks. 
Work in a biology lab. 
Perform jazz or tap dance. 
Help conduct a group therapy session. 
Market a new line of clothing. 
Record rent payments. 
Set up and operate machines to make products. 
Invent a replacement for sugar. 
Sing in a band. 
Take care of children at a day-care center. 
Sell merchandise at a department store. 
Keep inventory records. 
Put out forest fires. 
Do laboratory tests to identify diseases. 
Edit movies. 
Teach a high-school class. 
Manage a clothing store. 
Stamp, sort, and distribute mail for an organization. 
If a job had been available in the week ending last Sunday, would 
you have been able to start within 2 weeks? 
1 Yes 
2 No 
All things considered, how satisfied are you with your current 
job? Would you say you are: 
1 — Extremely satisfied 
2 - Satisfied 
3 — Neither satisfied nor dissatisfied 
4 — Dissatisfied 
5 - Extremely dissatisfied 
What occupation do you plan on pursuing? Please select a broad 
occupation category on this page and select more specific 
occupation groups and job titles on the following pages. 
Occupation: List of 10 major groups (Managers through Armed 
Forces) 
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Table B-2._ ESO Noncognitive Battery — Career Interest and Intentionality (CIJ)—Continued 


Question Description 

(190) CIL 90 — continued 
Managers List2 Managers Subcategories 
Professionals List2 Professionals Subcategories (6) 


Professionals — Business _ Administration List 
Professionals — Health List2 
Professionals — ICT List2 
Professionals — Legal-Social_ Cultural List2 
Professionals — Science _ Engineering List2 
Professionals — Teaching List2 
Technicians Associates List2 | Technicians and Associate Professionals Subcategories (5) 
Technicians _ Associates - ICT List2 
Technicians _ Associates - Health List2 
Tech _ Assoc - Science _ Engineering List2 
Tech Assoc - Legal Social Cultural List2 
Tech _ Assoc - Business _ Admin List2 
Clerical Support Workers List2 Clerical Support Workers Subcategories 
Services _ Sales Workers List2 Services and Sales Workers Subcategories 


Skilled Ag-Forestry_ Fish Skilled Agricultural, Forestry, and Fishery Workers 
Workers List2 Subcategories 

Crafts Related Trades Craft and Related Trades Workers Subcategories (5) 
Workers List2 


Crafts _ Rel Trades Workers - Building _ Rel List2 
Crafts _ Rel Trades Workers - Elec _ List2 

Crafts Rel Trades Workers - Food WW OtherList2 
Crafts Rel Trades Workers - Hand Print List2 
Crafts Rel Trades Workers - Met Mac Rel List2 
Plant _ Machine Operators _ Assemblers List2 
Elementary Occupations List2 

Armed Forces Occupations List2 


CIL 91 Are you planning on going to a further training, development, or 
educational program? 

CIL 92 What type of training, development, or educational program do 
you plan on attending? 

CIL 93 In what field of education will you attend a training, 


development, or educational program? 


End Survey 
ee ee ee es 
SOURCE: U.S. Department of Education, National Center for Education Statistics, PISA Young Adult Follow-up Study (PISA 
YAFS), 2016. 
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Table B-3. 


ESO Noncognitive Battery — Subjective Well-Being and Health (SWBH) 


Item # 


SWB&H_07 
SWB&H_08 
SWB&H_09 


SWB&H_11 


SWB&H 22 
SWB&H 23 
SWB&H_ 24 
SWB&H 25 
SWB&H 26 
SWB&H 27 
SWB&H_ 28 
SWB&H 29 
SWB&H 30 
SWB&H 37 


SWB&H_42 


SWB&H_46 


Concept/Construct 


Life evaluation - 
Satisfaction with life 
Life evaluation - 
Satisfaction with life 
Life evaluation - 
Satisfaction with life 
Life evaluation - 
Satisfaction with life 


Affect 
Affect 
Affect 
Affect 
Affect 
Affect 
Affect 
Affect 


Affect 
Subjective health 


indicator 


Objective health 
indicator 


Objective health 
indicator 


Item 

<Scale Stem> Below are statements that you may agree or 
disagree with. Using the scale, please indicate your agreement 
with each item. Please be open and honest in your responses. 


Scale: Strongly Disagree//Disagree//Slightly Disagree//Slightly 
Agree//Agree//Strongly Agree 
In most ways my life is close to my ideal. 


The conditions of my life are excellent. 
I am satisfied with my life. 
If I could live my life over, I would change almost nothing. 


<Scale Stem> Below is a list of words describing different 
emotions. Please indicate the extent to which you have felt this 
way over the past week. 


Scale: Very slightly or Not at all//A little//Moderately// 
Quite a bit//Extremely 

Distressed 

Excited 

Upset 

Happy 

Proud 

Ashamed 

Nervous 

Active 

Frustrated 

Would you say your health now is: 


Scale: Very Poor//Poor//Fair//Good//Very Good//Excellent 
<Item Stem>The following questions will ask about various 
health behaviors. Use the number keys to type your answer. 
Please provide a numeric value between [lower limit] and 
[upper limit]. 

During the past month, how would you rate your overall sleep 
quality? 


Scale: Very Bad//Fairly Bad//Fairly Good//Very Good 
During the past month, on average how many hours of actual 
sleep did you get at night? 


Scale: 
e Less than 7 hours 
e 7to 9 hours 
More than 9 hours 


See note at end of table. 
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Table B-3. ESO Noncognitive Battery — Subjective Well-Being and Health (SWBH)—Continued 


Item # 
SWB&H_ 48 


SWB&H_49 


SWB&H_50 


SWB&H_51 


SWB&H 54 


SWB&H_57 


SWB&H_58 


SWB&H_59 


SWB&H_60 


Concept/Construct 


Objective health 
indicator 


Objective health 
indicator 


Objective health 
indicator 


Objective health 
indicator 


Objective health 
indicator 


Objective health 
indicator 


Objective health 
indicator 


Objective health 
indicator 


Objective health 
indicator 


Item 
In a typical week, on how many days do you eat fruits? 


Scale: 0//1//2//3//4//5//6//7 
How many servings of fruit do you eat on one of those days? 


Scale: 0//1//2//3//4//more than 4 
In a typical week, on how many days do you eat vegetables? 


Scale: 0//1//2//3//4//5//6//7 
How many servings of vegetables do you eat on one of those 
days? 


Scale: 0//1//2//3//4// more than 4 
Do you currently smoke any tobacco products, such as 
cigarettes, cigars, or pipes? 


Scale: No//Yes, sometimes//Yes, daily 
In a typical week, on how many days do you engage in 
moderate physical activities (e.g., walking, climbing stairs)? 


Scale: 0//1//2//3//4//5//6//7 
How much time did you spend on moderate physical activities 
on a typical day? 


Scale: Minutes (---) 

In a typical week, on how many days do you engage in 
vigorous physical activities (e.g., running, cycling, team 
sports)? 


Scale: 0//1//2//3//4//5//6//7 
How much time did you spend on vigorous physical activities 
on a typical day? 


Scale: Minutes (---) 


SE ee a ee ee ee ee ee ee ee ee 
SOURCE: U.S. Department of Education, National Center for Education Statistics, PISA Young Adult Follow-up Study (PISA 


YAFS), 2016. 
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Table B-4. ESO Noncognitive Battery — Skill Use (WST) 


Variable 

WTST_02 
WTST_35 
WTST_36 
WTST 37 


WTST_38 
WTST_39 
WTST_40 
WTST_41 
WTST_42 
WTST_43 
WTST_44 
WTST_45 
WTST_46 
WTST_47 
WTST_48 
WTST_49 
WTST_50 
WTST 51 
WTST_52 
WTST_53 


WTST_54 
WTST_55 


WTST_56 
WTST_57 
WTST_58 
WTST_59 
WTST_60 
WTST 61 
WTST 62 
WTST 63 


WTST_64 
WTST_65 


Question 


Have you ever been employed? 

In your job, how often do you usually read directions or instructions? 

In your job, how often do you usually read letters, memos, or e-mails? 

In your job, how often do you usually read articles in newspapers, magazines, or 
newsletters? 

In your job, how often do you usually read articles in professional journals or scholarly 
publications? 

In your job, how often do you usually read books? 

In your job, how often do you usually read manuals or reference materials? 

In your job, how often do you usually read bills, invoices, bank statements, or other 
financial statements? 

In your job, how often do you usually read diagrams, maps, or schematics? 

In your job, how often do you usually write letters, memos, or e-mails? 

In your job, how often do you usually write articles for newspapers, magazines, or 
newsletters? 

In your job, how often do you usually write reports? 

In your job, how often do you usually fill in forms? 

In your job, how often do you usually calculate prices, costs, or budgets? 

In your job, how often do you usually use or calculate fractions, decimals, or 
percentages? 

In your job, how often do you usually use a calculator - either handheld or computer 
based? 

In your job, how often do you usually prepare charts, graphs, or tables? 

In your job, how often do you usually use simple algebra or formulas? 

In your job, how often do you usually use more advanced math or statistics such as 
calculus, complex algebra, trigonometry, or use of regression techniques? 

For this question, please consider your current job. If you are not currently employed, 
please consider your most recent job. Do you use a computer at work? 

In your job, how often do you usually Use e-mail? 

In your job, how often do you usually Use the Internet in order to better understand 
issues related to your work? 

In your job, how often do you usually conduct transactions on the Internet, for example, 
buying or selling products or services, or banking? 

In your job, how often do you usually use spreadsheet software, for example, Excel? 
In your job, how often do you usually use a word processor, for example, Word? 

In your job, how often do you usually use a programming language to program or write 
computer code? 

In your job, how often do you usually participate in real-time discussions on the Internet, 
for example, online conferences, or chat groups? 

For this question, please consider your current job. If you are not currently employed, 
please consider your most recent job. What level of computer use is needed to perform 
your job? 

Do you think you have the computer skills you need to do your job well? 

Has a lack of computer skills affected your chances of being hired for a job or getting a 
promotion or pay raise? 

In everyday life, how often do you usually read directions or instructions? 

In everyday life, how often do you usually read letters, memos, or e-mails? 


See note at end of table. 


PISA YAFS Technical Report and User Guide B-8 


Table B-4. ESO Noncognitive Battery — Skill Use (WST)—Continued 


Variable 
WTST_66 


WTST_67 
WTST_68 
WTST_69 
WTST_70 
WTST_71 
WTST_72 
WTST_73 
WTST_74 
WTST_75 
WTST_76 
WTST_77 
WTST_78 
WTST_79 
WTST_80 
WTST 81 
WTST_82 
WTST_83 
WTST_84 
WTST_85 
WTST_86 
WTST_87 
WTST_88 
WTST_89 


WTST_90 


Question 


In everyday life, how often do you usually read articles in newspapers, magazines, or 
newsletters? 

In everyday life, how often do you usually read articles in professional journals or 
scholarly publications? 

In everyday life, how often do you usually read books, fiction or nonfiction? 

In everyday life, how often do you usually read manuals or reference materials? 

In everyday life, how often do you usually read bills, invoices, bank statements, or other 
financial statements? 

In everyday life, how often do you usually read diagrams, maps, or schematics? 

In everyday life, how often do you usually write letters, memos, or e-mails? 

In everyday life, how often do you usually write articles for newspapers, magazines, or 
newsletters? 

In everyday life, how often do you usually write reports? 

In everyday life, how often do you usually fill in forms? 

In everyday life, how often do you usually calculate prices, costs, or budgets? 

In everyday life, how often do you usually use or calculate fractions, decimals, or 
percentages? 

In everyday life, how often do you usually use a calculator - either handheld or computer 
based? 

In everyday life, how often do you usually prepare charts, graphs, or tables? 

In everyday life, how often do you usually use simple algebra or formulas? 

In everyday life, how often do you usually use more advanced math or statistics such as 
calculus, complex algebra, trigonometry, or use of regression techniques? 

Have you ever used a computer? 

Do you use a computer in your everyday life now (outside work)? 

In your everyday life, how often do you usually use e-mail? 

In your everyday life, how often do you usually use the Internet in order to better 
understand issues related to, for example, your health or illnesses, financial matters, or 
environmental issues? 

In your everyday life, how often do you usually do you conduct transactions on the 
internet, for example, buying or selling products or services, or banking? 

In your everyday life, how often do you usually use spreadsheet software, for example, 
Excel? 

In your everyday life, how often do you usually use a word processor, for example, 
Word? 

In your everyday life, how often do you usually use a programming language to program 
or write computer code? 

In your everyday life, how often do you usually participate in real-time discussions on 
the Internet, for example, online conferences or chat groups? 


a  , —_ | 
SOURCE: U.S. Department of Education, National Center for Education Statistics, PISA Young Adult Follow-up Study (PISA 


YAFS), 2016. 
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Appendix C. List of Variables for Population Modeling 


Variable Name 


SCHOOLID 


STO1Q01 
ST04Q01 
STOSQO1 
STO7Q01 
STO7Q02 
ST07Q03 
STO8Q01 
STO9IQO1 
ST115Q01 
ST13Q01 
ST14Q01 


ST14Q02 
ST14Q03 
ST14Q04 
ST15QO1 
ST17Q01 
ST18Q01 


ST18Q02 
ST18Q03 
ST18Q04 
ST19Q01 
ST20001 
ST20Q02 
ST20Q03 
ST21001 
ST25Q01 
ST26Q01 
ST26Q02 
ST26Q03 
ST26Q04 
ST26Q05 
ST26Q06 
ST26Q07 
ST26Q08 
ST26Q09 
ST26Q10 
ST26Q11 
ST26Q12 
ST26Q16 
ST27002 
ST27004 


See note at end of table. 
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Source 
PISA 2012 


PISA 2012 
PISA 2012 
PISA 2012 
PISA 2012 
PISA 2012 
PISA 2012 
PISA 2012 
PISA 2012 
PISA 2012 
PISA 2012 
PISA 2012 


PISA 2012 
PISA 2012 
PISA 2012 
PISA 2012 
PISA 2012 
PISA 2012 


PISA 2012 
PISA 2012 
PISA 2012 
PISA 2012 
PISA 2012 
PISA 2012 
PISA 2012 
PISA 2012 
PISA 2012 
PISA 2012 
PISA 2012 
PISA 2012 
PISA 2012 
PISA 2012 
PISA 2012 
PISA 2012 
PISA 2012 
PISA 2012 
PISA 2012 
PISA 2012 
PISA 2012 
PISA 2012 
PISA 2012 
PISA 2012 


Variable Info 

School ID 7-digit (region ID + stratum ID + 3-digit 
school ID) 

International Grade 

Gender 

Attend pre-school or kindergarten 

Repeat — grades 1-6 

Repeat — grades 7-9 

Repeat — grades 10-12 

Truancy — Late for School 

Truancy — Skip whole school day 

Truancy — Skip classes within school day 
Mother Highest Schooling 

Mother Qualifications — Master’s, doctoral, or prof. 
degree 

Mother Qualifications — Bachelor’s degree 
Mother Qualifications — Associate’s degree 
Mother Qualifications — Voc/Tech cert. after HS 
Mother Current Job Status 

Father Highest Schooling 

Father Qualifications — Master’s, doctoral, or prof. 
degree 

Father Qualifications — Bachelor’s degree 
Father Qualifications — Associate’s degree 
Father Qualifications — Voc/Tech cert. after HS 
Father Current Job Status 

Country of Birth International — Self 

Country of Birth International — Mother 
Country of Birth International — Father 

Age of arrival in United States 

International Language at Home 

Possessions — desk 

Possessions — own room 

Possessions — study place 

Possessions — computer 

Possessions — software 

Possessions — Internet 

Possessions — literature 

Possessions — poetry 

Possessions — art 

Possessions — textbooks 

Possessions — technical reference books 
Possessions — dictionary 

Possessions — high-speed internet connection 
How many — televisions 

How many — cars 
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Variable Name 


ST27005 
ST28001 
ST37001 
ST37Q02 
ST37Q03 


ST37Q04 


ST37Q05 
ST37Q06 
ST37Q07 
ST37Q08 


ST81001 
ST81Q02 
ST81003 


ST81004 
ST81Q05 


ST87Q01 
ST87Q02 
ST87003 
ST87Q04 
ST87Q05 
ST87Q06 
ST87Q07 
ST89Q05 
ST94Q05 


ST94Q06 
ST94Q09 


ST94Q10 
ST94Q14 


COBN F 
COBN M 
COBN S 
ESCS 
hisced 
LANGN 
PVIMATH 
PVIREAD 
PV1SCIE 
RACETHC 


See note at end of table. 
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Source 


PISA 2012 
PISA 2012 
PISA 2012 
PISA 2012 
PISA 2012 


PISA 2012 


PISA 2012 
PISA 2012 
PISA 2012 
PISA 2012 


PISA 2012 
PISA 2012 
PISA 2012 


PISA 2012 


PISA 2012 


PISA 2012 
PISA 2012 
PISA 2012 
PISA 2012 
PISA 2012 
PISA 2012 
PISA 2012 
PISA 2012 


PISA 2012 


PISA 2012 


PISA 2012 
PISA 2012 


PISA 2012 
PISA 2012 
PISA 2012 
PISA 2012 
PISA 2012 
PISA 2012 
PISA 2012 
PISA 2012 
PISA 2012 
PISA 2012 


Variable Info 


How many — rooms bath or shower 

How many books at home 

Maths Self-Efficacy — Using a Train Timetable 
Maths Self-Efficacy — Calculating TV Discount 
Maths Self-Efficacy — Calculating Square Feet of 
Tiles 

Maths Self-Efficacy — Understanding Graphs in 
Newspapers 

Maths Self-Efficacy — Solving Equation 1 
Maths Self-Efficacy — Distance to Scale 

Maths Self-Efficacy — Solving Equation 2 
Maths Self-Efficacy — Calculate Petrol 
Consumption Rate 

Disciplinary Climate — Students Do Not Listen 
Disciplinary Climate — Noise and Disorder 
Disciplinary Climate — Teacher Has to Wait Until 
its Quiet 

Disciplinary Climate — Students Do Not Work 
Well 

Disciplinary Climate — Students Start Working 
Late 

Sense of Belonging - Feel Like Outsider 

Sense of Belonging — Make Friends Easily 
Sense of Belonging — Belong at School 

Sense of Belonging — Feel Awkward at School 
Sense of Belonging — Liked by Other Students 
Sense of Belonging — Feel Lonely at School 
Sense of Belonging — Feel Happy at School 
Attitude toward School — Trying Hard is Important 
Openness for Problem Solving — Can Handle a Lot 
of Information 

Openness for Problem Solving — Quick to 
Understand 

Openness for Problem Solving — Seek 
Explanations 

Openness for Problem Solving — Can Link Facts 
Openness for Problem Solving — Like to Solve 
Complex Problems 

Country of Birth National Categories — Father 
Country of Birth National Categories — Mother 
Country of Birth National Categories — Self 
Index of economic, social and cultural status 
Highest educational level of parents 

Language at home (3-digit code) 

Plausible value | in mathematics 

Plausible value | in reading 

Plausible value | in science 

NAT/Derived Race/Ethnicity (collapsed) 
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Variable Name 


SC03Q01 
FRPL 
SC54A01 
PUBPRIV 
B_Q02a 
B_Q02bUS 
B_Q02c 
B_Q04a 

B Ql2a 
B_Q12c 
B_Ql2e 

B Ql2g 
B_Q26a 
B_Q26b 
B_Q27aUSX 
B_Q27cUSX 
B_Q29aUSX 
BQ Q4 

BQ Q6 

BQ Q8 


Satisfaction_Life_ Label 
Positive _Affect_Label 
Negative_Affect_Label 


Subj Health Label 


SkillUse_Reading home 
SkillUse_Reading_work 
SkillUse_ Writing home 
SkillUse_ Writing work 


SkillUse_Numeracy_ 
home 
SkillUse_Numeracy_ 
work 
SkillUse_ICT_home 


SkillUse_ICT_work 
SeekJob Group 
SeekTraining Group 
SelfEfficacy_Group 


TakeActive Steps Group 


SWBH 37 


Source 


PISA 2012 

PISA 2012 

PISA 2012 

PISA 2012 

Westat administered 
Westat administered 
Westat administered 
Westat administered 
Westat administered 
Westat administered 
Westat administered 
Westat administered 
Westat administered 
Westat administered 
Westat administered 
Westat administered 
Westat administered 
E&S Online 

E&S Online 

E&S Online 

E&S Online 

E&S Online 

E&S Online 

E&S Online 

E&S Online 

E&S Online 

E&S Online 

E&S Online 

E&S Online 


E&S Online 
E&S Online 
E&S Online 
E&S Online 
E&S Online 
E&S Online 


E&S Online 
E&S Online 


Variable Info 


School Location 

NAT\Pct Free/reduced lunch (categorized) 
NAT\Pct students different first language 
NAT\Public/Private school indicator 
Currently studying 

Ed. Level (recode consistent with PIAAC) 
Area of study, Category 

Study last 12 months 

Last 12 months, distance ed. 

Last 12 months, job training 

Last 12 months, seminars 

Last 12 months, private training 

Last 12 months, wanted to participate 

Last 12 months, reasons not in ed. 

During 12 months, improve skills 

During 12 months, HS equivalency 

During 12 months, apprenticeship 

Born in country 

Language first learned and still understand 
Employment status 

Classification of satisfaction with respondent’s life 
Classification of experience of positive emotions 
Classification experience of negative emotions 
Self-assessment of health categorized 
Frequency of reading skill use at home 
Frequency of reading skill use at work 
Frequency of writing skill use at home 
Frequency of writing skill use at work 
Frequency of numeracy skill use at home 


Frequency of numeracy skill use at work 


Frequency of information and communication 
technology (ICT) skill use at home 

Frequency of ICT skill use at work 

Intention to find a new job 

Intention to seek additional training 

Personal intention and motivation for pursuing a 
new job 

Actions and steps taken in pursuit of a new job 
Self-assessment of health, specific 


Bee Ee SE eee ee eee SS eee eee ee eee eee Oe Eee eee 
SOURCE: U.S. Department of Education, National Center for Education Statistics, PISA Young Adult Follow-up Study (PISA 
YAFS), 2016. 
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Appendix D. PISA YAFS 2016 Recruiting Materials 
Exhibit D-1. Text of PISA Young Adult Follow-up Study NCES initial information letter 


U.S. DEPARTMENT OF EDUCATION 
INSTITUTE OF EDUCATION SCIENCES 


NATIONAL CENTER FOR EDUCATION STATISTICS 


Dear PISA participant, 


My name is Dan McGrath and I am the Director of the International 
Activities Program at the National Center for Education Statistics. In the fall 
of 2012 at your school, you participated in the Program for International 
Student Assessment (PISA), an international study of 15-year-olds’ 
mathematics, science, and reading skills, implemented by the U.S. 
Department of Education. At that time, you provided us with your contact 
information. 


We are contacting you to let you know that you will be invited to participate 
in the PISA Young Adult Follow-Up Study that we will conduct in 2015 to 
collect information about your education and work experiences, and your 
skills as a young adult. You will receive $50 once you complete the study in 
2015. 


For now, we need to verify your contact information. Please complete the 
attached Contact Information Update Form and return it to us in the 
enclosed envelope. Information collected on the form will be used only 
for the purposes of this study. 


We will be back in touch with you over the coming months to provide more 
information about the study. If you have any questions in the meantime, 
you may call the study’s toll-free number (1-855-604-1519), email at 
YAFollow-up@westat.com or visit the Young Adult Follow-up Study website 
at http://nces.ed.gov/surveys/pisa/followup.asp. 


We thank you in advance for your cooperation in this important research. 


Sincerely, 
Laut fp MB acte 


Daniel J. McGrath 

Director, International Activities Program 
National Center for Education Statistics 
Institute of Education Sciences 

U.S. Department of Education 
Telephone: (202) 502-7426 

Email: Daniel.McGrath@ed.gov 


WASHINGTON. D.C. 20006 
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Exhibit D-2. Text of PISA Young Adult Follow-up Study NCES contact information update form 


a | PISA Young Adult Follow-up Study 


Contact Information Update Form 


Please use a black or blue pen to complete this form. Mark XJ to indicate your answer. 
If you want to change your answer, darken the box §@ and mark the correct answer. 


What is your preferred method of being contacted? (Please mark all that apply.) 
OMail [JHomePhone [CellPhone [] Email 
(1 Request to receive communications by Text Message 


Please review the information below and make any updates in the boxes to the right. If all of the information is 
correct, please mark the "All information is up to date." box at the bottom. 


Information on File Updated Information 


1 No email address. 


Home Phone: [TT] = aaa = EESE O1None. 
Cell Phone: [TT] - aan - [LLU] None. 


Name (first, middle, last): 
Address 1: 
Address 2: 


City: 


All information is up to date. 


In the next box, please review the information you previously provided on a family member or friend who would 
know how to contact you two years from now. If all of the information is correct, please mark the "All information 
is up to date." box at the bottom. 


Information on File Updated Information 


No email address. 


Home Phone: Bee = [TT] = [TTT] O1None. 
Cell Phone: Ea - Gis - RSE ONone. 


Name (first, middle, last): 
Address 1: 
Address 2: 


City: 


Relationship to you: 
O Parent 1 Grandparent OJ Aunt/Uncle 
OBrother/Sister [Friend 0 Other 


All information is up to date. 


Thank you for completing the form. Please return it in the postage-paid envelope provided or mail it to: 
Young Adult Follow-up Study 
1600 Research Blvd 
Room RA 1164 
Rockville, MD 20850 
Toll-free number for questions or assistance 1-855-604-1519 or email YAFollow-up@westat.com. 


Data collected may be used only for statistical purposes and may not be disclosed, or used, in identifiable form for any other purpose (20 U.S.C., 
B § 9573). The U.S. Office of Management and Budget has approved this data collection under OMB 1850-0900. Approval expires 11/30/2016. || 
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Exhibit D-3. 


What is the PISA Young Adult 
Follow-Up Study? 


The PISA Young Adult Follow-Up Study (YAFS) 

is a survey of U.S. students who participated in 

the Program for International Student Assessment 
(PISA) in 2012. This is the first study of its kind to 
be conducted in the United States and will provide 
valuable information about how students’ skills 
measured by PISA at age 15 relate to education and 
work experiences and key skills in young adulthood. 
The study is voluntary, but only students who 
participated in PISA 2012 and provided contact 
information are invited to participate, making your 
participation essential for the study’s success. 


Why are you contacting me? 


You provided contact information after completing the 
PISA 2012 assessment so that the National Center for 
Education Statistics (NCES) could contact you for this 
follow-up study. You were also contacted to register 

on the PISA Young Adult Follow-up Study participant 
website. Only students that participated in PISA 

2012 and provided contact information are eligible to 
participate in the study. There are no replacements or 
other sources for the information that you will provide. 
As a token of our appreciation, upon completion of the 
online survey, you will receive a $50 incentive. 


‘le 


NATIONAL CENTER ror 
EDUCATION STATISTICS 


Institute of Education Sciences 


PISA Young Adult Follow-up Study information flyer 


Why is this study important? 


Conducting this follow-up study with PISA participants 
provides a unique opportunity to learn more about the 
relationship between students’ skills at age 15 and their 
education and work experiences as they enter adulthood. 
This information will help us better-understand the various 
paths (e.g. college, workforce, vocational training, etc.) 
U.S. students have taken after high school. 


What do I need to do now? 


The invitation letter accompanying this information sheet 
is your invitation to take the survey. Keep the enclosed cash 
card in a safe place. The card is not active until after you 
complete the survey. In a few days, you will be notified 
via email that the survey is open and to visit YAFStudy.org 
to complete the survey. Once you are notified, log in and 
complete the survey. Upon completion, your cash card 
will be activated and you can use it. 


How do | find out more? 
Visit us online at www.yafstudy.org or 


http://nces.ed.gov/surveys/ pisa/followup.as 
E-mail us at PISAFollowup@westat.com 
Call us at 1-855-604-1519. 


Young Adult Follow-Up Study 


<)> AYRES 


0.M.B. No. 1850-0900 Expiration date: 03/31/2018 
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Exhibit D-4. Text of PISA Young Adult Follow-up Study NCES participant website registration letter 


U.S. DEPARTMENT OF EDUCATION 
INSTITUTE OF EDUCATION SCIENCES 


NATIONAL CENTER FOR EDUCATION STATISTICS 


Dear [FIRST NAME], 


On behalf of NCES and the U.S. Department of Education, I want to thank you for 
responding to our previous information update requests for the PISA Young Adult 
Follow-Up Study (YAFS). Your participation is vital in helping us better understand 
how well prepared U.S. young adults are for the global economy. 


PISA YAFS will take place later in 2015. For now, we ask that you log in to the 
PISA YAFS participant website using the credentials below to verify your contact 
information so we can contact you when the survey is officially open. The 
information you provide will also be used to send the cash card that will be loaded 
with $50 when you complete the survey. 


You must log in to the PISA YAFS participant website to take part in the study. 
Website: <INSERTHYPERLINK> 


Initial Username: <USERNAME> | 


Initial password: <PWD> | 


Your username and password are unique to you, so please do not share with 
anyone else. Once you have logged in to the website and created a new user 
name and password you will receive a confirmation email. Keep your new user 
name and password handy because you will need them again when it is time to 
complete the survey. 


Before the survey opens later in 2015, we will mail your cash card and 
instructions on how to complete the survey to the mailing address you provide on 
the website. 


If you have any questions or need any assistance logging in to the website, you 
may call the study’s toll-free number (1-855-604-1519) or email 
PISAFollowup@westat.com 


We value your time and participation in the PISA Young Adult Follow-Up Study. 


Sincerely, 
CL) 


Por 


{ f WRucte 


# 
Daniel J. McGrath 
Director, International Activities Program 
National Center for Education Statistics 
Institute of Education Sciences 
U.S. Department of Education 
Telephone: (202) 502-7426 


Email: Daniel.McGrath@ed.gov 


WASHINGTON. D.C. 20006 
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Exhibit D-5. Text of PISA Young Adult Follow-up Study participant website registration confirmation 


& >BYAFS 


Young Adult Follow-Up Study 


Dear <First name> 
Welcome to the PISA Young Adult Follow-Up Study! 


Thank you for completing your online account setup. Your participation in the PISA YAFS is important, as 
you are a unique source of information about how students’ experiences and skills at age 15 relate to 
their experiences and skills in young adulthood. This information will inform discussions about college 
and career-readiness and successful transitions from high school to college and the workforce. 


Your new user name is: <USER NAME> 
YAFS Participant Website: www.YAFStudy.org 


Please remember to keep your password handy. Answers to the security questions you selected when 
registering are required if you forget your password and need to reset it. Please send an email or call the 
PISA YAFS support staff at PISAFollowup@westat.com or 1-855-604-1519 if you need any assistance 
logging in. 
What to expect next: 
Before the survey opens later in 2015, you will be sent an email with instructions to access the 
survey, and a FedEx package with your cash card. 
Once you receive the FedEx package, please log in to <website hyperlink> and complete the 


survey. 
Once you have completed the survey, your cash card will be loaded with $50.00. 


For more information on the study, click here to visit the NCES YAFS study website. 


If you have any questions or need assistance while completing any of these steps, please contact us 
using the email link or phone number above. 


@ 
@ 
e | e S NATIONAL CENTER For 
e EDUCATION STATISTICS 


Institute of Education Sciences 
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Exhibit D-6. Text of PISA Young Adult Follow-up Study NCES primer e-mail: pilot and main study 


Subject: The PISA Follow-up Study is launching in [Month]— Update and next steps 
Dear <SP Full Name>, 


We are pleased to announce that the first wave of the PISA Young Adult Follow-up Study (YAFS) is 
launching in the beginning of December! 


You are currently registered to participate in the first wave of YAFS. 


Prior to the study launch in [Month], you will receive a FedEx package with login instructions and 
your unloaded cash card. 


Please log in to www. Y AFStudy.org and update your address and phone number by [Date] so 
that we have the location where you want to receive your package. 


Your username is: <User Name> 


Your password is unknown to us for security. If you’ ve forgotten your password you can use the 
“Forgot password?” link on the home page to retrieve and reset it. 

After completing the study, you will be asked to enter the last four digits of the cash card to activate it 
and receive your $50.00 incentive. 


Part of participating in the study includes completing the Education and Skills Online (ESO) 
assessment. You can complete the study from any laptop or desktop computer with an internet 
connection, though you must use the Firefox browser. No other browser will work and neither will 
tablets or cellphones. 


To download and install Firefox (about 90MB) go to: 
https://www.mozilla.org/en-US/firefox/new/?scene=2#download-fx 


If you have questions or need help, please contact us at e-mail address or call us at 1-800-number. 


Thank you for your time and participation in the first-ever PISA Young Adult Follow-up Study! 
The PISA YAFS Team 
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Exhibit D-7. PISA Young Adult Follow-up Study NCES invitation letter 


Dear <first name> <Last name>, 


My name is Dan McGrath and I am the Director of the International Activities Program at the National 
Center for Education Statistics. | am writing to invite you to participate in the PISA Young Adult 
Follow-Up Study. As we have described in previous communications, this is an invitation-only online 
survey that collects information about your education and work experiences and your skills as a young 
adult. The survey should take 50 minutes on average but no more than 2 hours to complete. Once you 
complete the online survey, $50 will be loaded on the enclosed cash card. You may use this cash 
card anywhere MasterCard is accepted. 


We will send you the web address and login information on [Month Day, Year], the day before the 
survey opens. In the meantime, please keep the enclosed cash card safe. 


If you have any questions or need help logging in when the survey opens, please call 1-855-604-1519 
(toll free), send an e-mail to YAFollow-up@westat.com, or visit the PISA Young Adult Follow-up 
Study website at https://nces.ed.gov/surveys/pisa/followup.asp. 


On behalf of the U.S. Department of Education, I want to thank you in advance for your cooperation 
in this important research project. 


Sincerely, 


Daniel J. McGrath 

Director, International Activities Program 
National Center for Education Statistics 
Institute of Education Sciences 

U.S. Department of Education 
Telephone: (202) 502-7426 

E-mail: Daniel.McGrath@ed.gov 
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Exhibit D-8. PISA Young Adult Follow-up Study respondent reminder 


Dear <First name> <Last Name>, 


There are only [X] days until the PISA Young Adult Follow-up Study begins! Remember, your 
participation is critical to the success of the study and we are counting on you to complete the survey. 
[Last week] you should have received a letter from Dan McGrath of the National Center for Education 
Statistics in the U.S. Department of Education inviting you to participate in the PISA Young Adult 
Follow-up Study. Your cash card was included with that letter. If you did not receive the invitation 
letter, or you have lost or misplaced your card, we can issue another one for you. Just call us at 
1-855-604-1519 (toll free) or e-mail us at YAFollow-up@westat.com and we can get a new card to 
you. Remember, the survey should take 50 minutes on average but no more than 2 hours to complete 
and the card will work after you complete the survey on [Month Day, Year]. 


Thanks for your participation! 


The PISA Young Adult Follow-up Study team 


NCES is authorized to conduct this study under the Education Sciences Reform Act of 2002 (ESRA 2002, 20 U.S.C., § 9543). Your 
participation is voluntary and the information you provide may be used only for statistical purposes and may not be disclosed or used, in 
identifiable form for any other purpose except as required by law (ESRA 2002, 20 U.S.C., § 9573). Individual responses will be combined 


with those from other participants to produce summary statistics and reports. NCES will securely maintain your contact information to locate 


you should we conduct another follow-up survey in the future. 
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Exhibit D-9. PISA Young Adult Follow-up Study announcement that survey is open 


Dear <First name> <Last Name>, 


The PISA Young Adult Follow-up Study is now open! You can access the survey through this link: 
[https://some link here]. Enter your login code to begin. Your login code 1s: 


[Login code here] 
You can also access the survey from your unique user account in the PISA Young Adult Follow-up 
Study website: https://www.YAFS.org. Once you are logged into the website, click the link [link 


name] and you can begin the survey. 


The survey should take 50 minutes on average but no more than 2 hours to complete. After you have 
completed the survey and you log out, your $50 cash card will be activated and you may use it. 


If you have any problems logging in to the survey or if you have other questions, please call us at 1- 
855-604-1519 (toll free) or e-mail us at YAFollow-up@westat.com and we can help you. 


Thanks for your participation! 


The PISA Young Adult Follow-up Study team 


NCES is authorized to conduct this study under the Education Sciences Reform Act of 2002 (ESRA 2002, 20 U.S.C., § 9543). Your 
participation is voluntary and the information you provide may be used only for statistical purposes and may not be disclosed or used, in 
identifiable form for any other purpose except as required by law (ESRA 2002, 20 U.S.C., § 9573). Individual responses will be combined 
with those from other participants to produce summary statistics and reports. NCES will securely maintain your contact information to locate 
you should we conduct another follow-up survey in the future. 
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Exhibit D-10._ PISA Young Adult Follow-up Study nonresponse e-mail/hardcopy — credentials sent 


Dear <first name> <Last name>, 


My name is Dan McGrath and I am the Director of the International Activities Program at the National 
Center for Education Statistics. | am writing to you about participating in our PISA Young Adult 
Follow-up Study. 


Our records indicate you have not yet logged into the survey. It looks like we sent you the web address 
login information, and unloaded cash card on [Month Day, Year]. Did you receive this package? If 
not, please [reply to this e-mail] [call us at 1-855-604-1519 (toll free) or send an e-mail to 
YAFollow-up@westat.com] and we will get another one to you immediately. 


As we have described in previous communications, this is an invitation-only online survey that 
collects information about your education and work experiences and your skills as a young adult. The 
survey should take 50 minutes on average but no more than 2 hours to complete. Once you complete 
the online survey, $50 will be loaded on your cash card. You may use this cash card anywhere 
MasterCard is accepted. Again, if you did not receive this card or have misplaced it, simply reply to 
this e-mail and we will get it to you immediately. 


If you have received your credentials but have not had the time to complete the survey, don’t worry. 
You have until MM/DD/YYYY to complete it. I will send you another reminder when the survey is 
close to ending so that you do not miss this opportunity to participate in this study. 


If you have any questions or need help logging in or accessing the internet, please call 1-855-604-1519 
(toll free) or send an e-mail to YAFollow-up@westat.com. 


On behalf of the U.S. Department of Education, I want to thank you in advance for your cooperation 
in this important research project. 


Sincerely, 


Daniel J. McGrath 

Director, International Activities Program 
National Center for Education Statistics 
Institute of Education Sciences 

U.S. Department of Education 
Telephone: (202) 502-7426 


E-mail: Daniel.McGrath@ed.gov 


Your participation is voluntary and the information you provide may be used only for statistical purposes and may not be disclosed or used, 
in identifiable form for any other purpose except as required by law (ESRA 2002, 20 U.S.C., § 9573). Individual responses will be combined 


with those from other participants to produce summary statistics and reports. NCES will securely maintain your contact information to locate 


you should we conduct another follow-up survey in the future. 
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Exhibit D-11. PISA Young Adult Follow-up Study partial complete 


Dear <First name> <Last Name>, 


Thank you for participating in the PISA Young Adult Follow-up Study! Your survey is almost 
complete, just a few more questions to go! For your responses to count, you need to complete the 
entire survey. Please log back in and complete the remainder of the survey. 


If you have any problems logging back in to the survey or if you have other questions, please call us at 
1-855-604-1519 (toll free) or e-mail us at YAFollow-up@westat.com and we can help you. 


Thanks for your participation! 


The PISA Young Adult Follow-up Study team 


Your participation is voluntary and the information you provide may be used only for statistical purposes and may not be disclosed or used, 
in identifiable form for any other purpose except as required by law (ESRA 2002, 20 U.S.C., § 9573). Individual responses will be combined 
with those from other participants to produce summary statistics and reports. NCES will securely maintain your contact information to locate 


you should we conduct another follow-up survey in the future. 
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Exhibit D-12._ PISA Young Adult Follow-up Study notification of survey closing 


Dear <first name> <Last name>, 


My name is Dan McGrath and I am the Director of the International Activities Program at the National 
Center for Education Statistics. | am writing to you about participating in our PISA Young Adult 
Follow-up Study. 


Our records indicate that you have not yet logged into the survey. This is your last chance to 
complete the survey and receive your $50! The survey will close on MM/DD/YYYY. Your 
participation in the PISA Young Adult Follow-up Study is valuable and important to the success of the 
survey. 


It looks like we sent you the web address login information, and unloaded the cash card on [Month 
Day, Year]. Did you receive this package? If not, please [reply to this e-mail] [call us at 1-855-604- 
1519 (toll free) or send an e-mail to YAFollow-up@westat.com] and we will get another one to you 
immediately. 


As we have described in previous communications, this is an invitation-only online survey that 
collects information about your education and work experiences and your skills as a young adult. The 
survey should take 50 minutes on average but no more than 2 hours to complete. Once you complete 
the online survey, $50 will be loaded on your cash card. You may use this cash card anywhere 
MasterCard is accepted. Again, if you did not receive this card or have misplaced it, simply reply to 
this e-mail and we will get it to you immediately. 


If you have any questions or need help logging in or accessing the internet, please call 1-855-604-1519 
(toll free) or send an e-mail to YAFollow-up@westat.com. 


On behalf of the U.S. Department of Education, I want to thank you in advance for your cooperation 
in this important research project. 


Sincerely, 


Daniel J. McGrath 

Director, International Activities Program 
National Center for Education Statistics 
Institute of Education Sciences 

U.S. Department of Education 
Telephone: (202) 502-7426 

E-mail: Daniel.McGrath@ed.gov 


Your participation is voluntary and the information you provide may be used only for statistical purposes and may not be disclosed or used, 
in identifiable form for any other purpose except as required by law (ESRA 2002, 20 U.S.C., § 9573). Individual responses will be combined 
with those from other participants to produce summary statistics and reports. NCES will securely maintain your contact information to locate 


you should we conduct another follow-up survey in the future. 
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Exhibit D-13. PISA Young Adult Follow-up Study thank you e-mail for completing the survey 


Dear <First Name <Last Name> 


On behalf of the U.S. Department of Education, I want to thank you for your participation in the PISA 
Young Adult Follow-up Study. This is the first study of its kind to be conducted in the United States. 
Through your participation in this important research project you provided a unique opportunity for 
the U.S. to better understand the educational and work experiences and the use of computers and the 
Internet in the daily lives of our nation. Your contributions are valued and appreciated. 


As a token of appreciation for your contributions to this important research, you will receive $50.00 
loaded onto the cash card you received. If you did not receive a cash card or have misplaced it, simply 
respond to this e-mail address and request a new one, or call 1-855-604-1519 (toll free). 

Again, thank you for your participation and cooperation throughout this study. 


Sincerely, 


Daniel J. McGrath 

Director, International Activities Program 
National Center for Education Statistics 
Institute of Education Sciences 

U.S. Department of Education 
Telephone: (202) 502-7426 

E-mail: Daniel.McGrath@ed.gov 
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